Cooling Challenges an Issue In Rackspace Outage

miller60 writes "If your data center's cooling system fails, how long do you have before your servers overheat? The shrinking window for recovery from a grid power outage appears to have been an issue in Monday night's downtime for some customers of Rackspace, which has historically been among the most reliable hosting providers. The company's Dallas data center lost power when a traffic accident damaged a nearby power transformer. There were difficulties getting the chillers fully back online (it's not clear if this was equipment issues or subsequent power bumps) and temperatures rose in the data center, forcing Rackspace to take customer servers offline to protect the equipment. A recent study found that a data center running at 5 kilowatts per server cabinet may experience a thermal shutdown in as little as three minutes during a power outage. The short recovery window from cooling outages has been a hot topic in discussions of data center energy efficiency. One strategy being actively debated is raising the temperature set point in the data center, which trims power bills but may create a less forgiving environment in a cooling outage."

Comments

This is number 3

[DuctTape]

This is actually Rackspace's number 3 outage in the past couple days. My company was only (!) affected by outages 1 and 2. My boss would have had a fit if number 3 would have taken us down for the third time.

Other publications have noted it was number 3, too.

DT

Re: This is number 3

[Kanuck54]

Our organization was only affected by outage #3. To my knowledge nobody was affected by all three outages; when Rackspace needed to choose which rows of servers to take offline to save the rest, they chose ones that hadn't gone down the day before. Tough choice, but they did the right thing, rather than waiting for everyone's servers to burn out.

Obviously this was a worst-case scenario, but they did handle it admirably. All hands were on deck all night, their chairman even posted a personal apology on our blog inside an hour of our post about the outage. It was good to know we weren't the only ones rushing to work in a panic late at night.

Still, it's an outright disaster when a data centre goes down. You would think a company that advises on redundant servers, redundant storage and redundant networking would recognize the importance of redundant cooling. They tell us this scenario was part of their disaster recovery plans; those plans simply didn't work as designed.

Let's just be grateful two hours of business was all that was lost.

Re:This is number 3

[jeffmeden]

This just goes to show that Rackspace, for all their good intentions, does not provide adequate redundancy for a truly 0-downtime operation.

The #1 observation people have is 'why werent the coolers on backup power?', and that's easy, backup power is very expensive and precious. They would need a completely separate backup generator system for the cooling, at least 50% as large as their current generator set. Not cheap. The cheap solution many use is 'chilled reserve', which is a thermal reserve of cold water/refrigerant to provide a cool source to the CRAC units. That, or you can go all out (as an unnamed company has done) and build your datacenter on top of a reservoir with over a hundred thousand gallons of water, the ultimate 'heat sink'.

The #2 observation people have is 'why weren't there alternate power routes coming into the facility?'. This, again, is a redundancy issue many top-notch data centers have accounted for, with multiple distinct power inputs and massive switchgear to allow either to fully power the facility, with automated switchover at a moment's notice.

Re:This is number 3

[JavaManJim]

Rackspace is doing this on the cheap and cheap. Their backup generators SHOULD provide a week's worth of backup.

Then as stated for #2 alternate power sources should be required. Rackspace fails.

GREAT BACKUP EXAMPLE. READ AND BE WISER! Especially you folks at Rackspace who will not hire older more experienced people. This is what you miss. Atlantic Richfield Oil and Gas (ARCO) set up a new data center many years ago in 1980. They located this in Plano TX. They decided that power backup was insufficient so ARCO paid 12 million dollars (1980 dollars) to run another electricity transmission line to their data center. This was a two or three mile line of those tall metal framework transmission towers. This can still be seen today if you travel Plano Parkway East of Coit. Google Earth at 33 00' 20.49"N, 96 44' 12.99"W. An unassuming building today. Also today you would locate on top of the 15 cable run (i.e. huge) dark fiber that goes north and south through Dallas.

Also ARCO had a fully configured but empty data center in Missouri to use if Plano ever suddenly wen. The data center lacked hardware but ARCO had an agreement with IBM to jump to the front of the mainframe production line if this disaster ever happened. Then it was a matter of applying backup tapes.

Rackspace should have considered optioning space at this old ARCO location which now has an insurance company there rather than where they are now.

Thanks,
Jim

Which only shows

[CaptainPatent]

If you want 100% uptime, it's important to have back up power for the cooling as well as the server systems themselves.
 
Is this really news?

Re:Which only shows

[_14k4]

Has anyone thought about putting data-centers in upper Canada / arctic regions? Just (honestly) curious.

Re:Which only shows

[Critical Facilities]

Exactly! The fact that these Chillers weren't on Emergency Generator Power is rookie mistake #1. All the generator power and UPS power in the world ain't gonna help if your Data Center gets too hot.

Re:Which only shows

[jandrese]

If you want 100% uptime (which is impossible, but you can put enough 9s in your reliability to be close enough), you need to have your data distributed across multiple data centers, geographically separate, and over provisioned enough that the loss of one data center won't cause the others to be overloaded. It's important to keep your geographical separation large because you never know when the entire eastern (or western) seaboard will experience complete power failure or when a major backhaul router will go down/have a line cut. Preferably each data center should get power from multiple sources if they can, and multiple POPs on the internet from each center is almost mandatory.

Re:Which only shows

[lb746]

I actually use a vent duct to suck in cold air from outside during the winter to help cool a server in my house. Originally I was more concerned with random object/bugs/leaves so I made it a closed system(like water cooling) to help protect the actual system. It works nicely, but only for about 1/3 or less of the year when the temperature is cold enough to make a difference. I've always wondered about a larger scale of something like this such as how the parent suggested servers in a colder/arctic region.

Re:Which only shows

[ByOhTek]

Or Siberia.

I was thinking the same thing.

AC is out? Crank open the vents and turn on the fans.

Admittedly it wouldn't work so well in the summer, but spring/winter/fall could be nice.

Re:Which only shows

[Ironsides]

Yes, actually. This was looked into by multiple companies during the late 90's. I'm not sure if any were ever built. I think one of the considerations as a byproduct was the savings of not having to run chillers with the cost of getting fibre and power laid to the facility.

Re:Which only shows

[Azarael]

Some data centers also have multiple incoming power lines (which hopefully don't have a single transformer bottle-neck). Anyway, I know for sure that at least one data center in Toronto had 100% uptime during the big August 2004 Blackout, so it is possible to prevent these problems.

Re:Which only shows

[Anonymous Coward]

Has anyone thought about putting data-centers in upper Canada / arctic regions? Just (honestly) curious.

There isn't any cheap high-speed fiber up there. Even if there was, the additional lag due to 5000 miles of fiber would be annoying, not to mention shipping & transport costs.

Re:Which only shows

[_14k4]

Well, being that the planet is (generally speaking) a sphere... you'll only really need at max, a D length piece of cable, no? :P What else are all of those holes to China I dug as a child good for?

Re:Which only shows

[blhack]

I think the problem is availability of power. When you are talking about facilities that consume so much power that, when built, their proximity to a power station is taken into account, you can't just slap one down at the poles and call it good. I would imagine that lack of bandwidth is a MAJOR issue as well..... ...one field where I think storing servers at the poles would be amazing is super computing. Supercomputers don't require the massive ammounts of bandwidth that webservers etc do. You send a cluster a chunk of data for processing, it processes it, and it gets sent back. For really REALLY large datasets (government stuff)...just fill a jet with hard-disks and have it to the server center in a few hours.

Re:Which only shows

[trolltalk.com]

In the winter, if you heat with electricity, you can basically run your computer for free, since its waste heat reduces the amount of heat needed to be generated by resistance heaters.

Re:Which only shows

[afidel]

It sounds like they DID have backup power for the cooling but that they switch over to backup power caused some problems. This isn't really all that unusual because cooling is basically never on UPS power so the transition to backup power may not go completely smoothly unless everything is setup correctly, tested, and there are no or little unusual circumstances during the switchover. I've seen even well designed systems have problems in the real world. One time we lost one leg of a triphase power system so the automatic transfer switch failed to flip over and startup the generator. The UPS realized it wasn't getting good power so it flipped over to battery power. Luckily the UPS sent out its notification and we were able to manually switch over to generator and get the cooling online, but there is almost no chance of it working in we only had 3 minutes to get things corrected.

Re:Which only shows

[ByOhTek]

An Athlon XP 2500+ with two HDDs and a GeForce 6600GT running 24x7 can give a 12x14 foot room a 10-15 degree temperature boost.

My computer room is quite toasty in the winter...

Re:Which only shows

[Bandman]

I'm not disagreeing, but if people really want 100% uptime, they're much better off investigating an infrastructure using GSLB or something similar, where a single geographically isolated event won't impact them

Re:Which only shows

[beavis88]

From a message to Rackspace customers:

"When generator power was established two chillers within the data center failed to start back up"

They had backup power for the chillers - but obviously, something didn't go quite right.

Re:Which only shows

[NickCatal]

I can't stress this enough. When I talk to people about hosting and they rely on 100% availability they NEED to go with geographically diverse locations. Even if it is a single backup somewhere you have to have something.

For example, Chicago's primary datacenter facility is in 350 E. Cermak (right next to McCormick Place) and the primary interconnect facility in that building is Equinix (which has the 5th and now 6th floors.) A year or so ago there was a major outage there (that mucked up a good amount of the internet in the midwest) when a power substation caught on fire and the Chicago Fire Department had to shut off power to the entire neighborhood. So the backup system started like it should, with the huge battery rooms powering everything (including the chillers) for a bit while the engineers started up the generators. Only thing is, the circuitry that controls the generators shorted out, so while the generators themselves were working, the UPS was working, the chillers were working, this one circuit board blew at the WRONG moment. And this isn't the only time this circuit has been used, they test the generators every few weeks.

Long story short, once the UPSes started running out of power the chillers started going, lights flickered, and for a VERY SHORT period of time the chillers went out before all of the servers did. Within a minute or two it got well over 100 degrees in that datacenter. Thank god the power cut out as quick as it did.

So yes, Equinix in that case did everything by the book. They had everything setup as you would set it up. It was no big deal. But something went wrong at the worst time for it to go wrong and all hell broke loose.

It could be worse, your datacenter could be hit by a tornado

Re:Which only shows

[l-ascorbic]

Norway is obviously the answer then. Bloody freezing, and loads of hydroelectric power.

Re:Which only shows

[russ1337]

I'm about to move to a colder (and more damp) environment (and an older house with wood floors), and have thought about putting my server and NAS in hallway cupboard, drawing cool air from under the house and venting it to the hallway.

Will have to see how big a hole I can cut (and repair) in the floor of the cupboard without the new landlord noticing....

Re:Which only shows

[Average]

Bandwidth is comparatively cheap to get somewhere. A few redundant loops of fiber... undersea if need be. Fiber does not suffer transmission losses in the way that sending electricity the other way would.

One fairly obvious location for this would be Labrador in Canada. Very well cooled. Absolutely lots of hydroelectric. Churchill Falls is huge. They lose half the energy sending it down to the US, but no one closer needs the power. Several major untapped hydro locations, too. Lots of land for approximately free.

Re:Which only shows

[SleptThroughClass]

Or instead of depending so much on chillers which draw a lot of power, buffer the cooling by having a large amount of chilled water as part of the system. Design it so if the chillers aren't working, the server room is still somewhat cooled by the pre-chilled water; circulation of water and air requires less power than refrigeration systems. I doubt most sites can store enough ice/chilled water to operate for hours. If that were the case, server rooms could be operating off of cold which was stored during cheaper evening electric rates.

Re:Which only shows

[scheme]

one field where I think storing servers at the poles would be amazing is super computing. Supercomputers don't require the massive ammounts of bandwidth that webservers etc do.

Supercomputing absolutely requires massive amounts of bandwidth. In a particle physics, detectors at places like LHC are generating 1-5 petabytes of data each year and this data needs to be sent out to centers and processed. Likewise, bioinformatics applications tend to generate lots of data (sequences, proteins, etc.) and this data needs to be shuttled around while it's being processed.

Re:Which only shows

[profplump]

Ice you can do; water doesn't have enough thermal capacity to be useful as a storage mechanism, unless you have a ludicrous amount of space to dedicate to chilled water storage. But even ice requires a fair amount of space, and the tanks themselves are not cheap. Ice management is a feature of almost any modern chiller system, it's just not terribly popular except in the largest installations.

And even with an ice system, you only get to turn off the compressor. You still need to run all the water pumps, including the big ones the move water from the basement ice storage to the primary heat exchanger, and all the smaller loops that run to air handling units. It's less power than running the compressor, but it's not like you wouldn't need the cooling system to be on the backup generator.

Re:Which only shows

[Critical Facilities]

This actually IS unusual. It's true that you would never put your cooling units on UPS power, but you would absolutely put them on generator power. Most people fail to differentiate between UPS power and Emergency Power. Having redundant utility feeds is great, but you have to have the generators in case both feeds fail.

It doesn't sound like the systems from your story were very well designed/maintained. If you were to lose 1 leg of your 3 phase power, the transfer switch would have gone to emergency position. If it didn't, someone's not doing their job (i.e. preventive maintenance, load testing, infrared scanning). Also, if you only had 3 minutes of battery time for your critical load, someone was not managing the load properly. Generally, you only need 15 - 30 seconds of battery time to make a transfer to generator, but to have ONLY 3 minutes worth is cutting it awfully close.

Re:Which only shows

[Critical Facilities]

They have this already. It's called "Economizer Mode". There are basically 2 types, water side economizer and air side economizer. The air side is similar to what you're doing, which is taking the raw, unconditioned air, filtering it, and using it to cool the space.

The other type is the Water Side Economizer which is probably going to be more prevalent in Data Centers. The way it works is, at a predetermined Outside Wetbulb Temperature, the cooling tower(s) speed up to cool the Condenser Water to a lower temperature than usual. Then, the air handlers/CRAC units switch the flow of the water from their condenser assemblies to a water coil. The air from the Data Floor releases its heat as it passes across the water coil, the water is returned to the cooling tower and the heat is ejected. Basically, you cool your data center with water that's cooled by evaporation and exposure to colder outside temperature rather than through mechanical refrigeration.

Re:Which only shows

[arth1]

Also the land of mountains and fjords, meaning that each mile of distance can equate to five miles or more of cable. Latencies is a bitch, and Norway isn't exactly centrally located to start with.
Not to mention that Norway, being one of the richest countries in the world, would be prohibitively expensive, with starting salaries around twice of those in the US, 37.5 hr work weeks, a minimum of 4 weeks of paid vacation, and very high taxes (including wealth tax, capital gain taxes, employer taxes and a 25% VAT (somewhat similar to sales tax)).
There are reasons why there's so few data centers in Norway, Iceland and Siberia.

Re:Which only shows

[Critical Facilities]

Hogwash.

Re:Which only shows

[LiquidCoooled]

Aren't most (all?) air conditioning systems ducted to the outside world?
In winter they do their job more efficiently.

Re:Which only shows

[Nos.]

Until this is a fire somewhere in the building and the fire department cuts ALL power. If its not geographically disperse, its not 100%.

Re:Which only shows

[autocracy]

Reading the article, they WERE on backup power. Emergency crews shut down the backup power to the chillers temporarily while they were working in the area, so the chillers had to start again. Cycling these big machines isn't instant.

Re:Which only shows

[spun]

Hmph. We have backup power for the cooling in our server room, but we had to deal with a fun little incident two weeks ago. Trane sent out a new HVAC monkey a month ago for routine maintenance. I was the one who let this doofus in, and let me tell you, he was a slack-jawed mouth-breathing yokel of tender years. He took one look at our equipment and said, I quote, "I ain't never seen nutin' like this'un before, hee-yuck!" I was a bit taken aback, but he seemed to go through all the proper motions.

Fast forward to three weeks ago. The temp is fine, but the humidity keeps going down. I tell management, but this is a state agency and everything around here takes three times as long as it should. For a state agency, that's outstanding, by the way. Anyway, noting gets done. Then we find out WHY the humidity is going down: seems the HVAC monkey didn't screw in the water bottle all the way and the entire 5 ton fills up with water, until it shorts out at 4 pm on a Friday afternoon and dumps water everywhere.

Well, we got our four emergency portable coolers in with little tubes leading out into the hall, the fans on, and the doors open right quick, but the temp still shot up to over 100 in under ten minutes. Well, I told hem something was up, and anyway, I'm on the VMware/BladeCenter server consolidation team, and this is just more of an argument to fund us better. But I guess the moral of the story is, don't let slack-jawed mouth-breathing yokels fix your mission critical systems.

Re:Which only shows

[TooMuchToDo]

So yes, Equinix in that case did everything by the book. They had everything setup as you would set it up. It was no big deal. But something went wrong at the worst time for it to go wrong and all hell broke loose.

As a previous customer in that facility who had equipment there, I can say that it was a HUGE deal. Equinix argues that their price premium (typically 3x what other datacenter facilities charge) is because you absolutely, positively won't ever go down because of them. After the outage you specified, as well as other problems they experienced, we moved a huge environment out to a datacenter in Oakbrook.

I'm not looking for 100% uptime, but don't rape me on datacenter space and tell me it's ever so fantastic, and then whine that there's nothing you could do when you go down for 12 hours.

Re:Which only shows

[Critical Facilities]

I realize that. But the article doesn't say that they were on backup GENERATOR power, but that they were on a redundant utility feed. Cycling these machines is not instant, I agree. However, the delay in restarting the chillers should not be such that you lose the floor. Also, the engineers in the building should have realized that their second utility supply was likely to be affected and gone to generator as a precaution until the repairs were made. It's better to be safe than sorry.

Re:Which only shows

[afidel]

Ok, I specifically said UPS power, as in it takes time to spinup the generators and switching from one source to the other does not always go perfectly in the real world. One factor is minimum cycle time on the compressors. The 3 minute time frame was from TFA which says that at a density of 5KVA per cabinet thermal shutdown can happen in 3 minutes due to thermal load.

Oh and as far as the one leg collapsing thing, yes we were VERY pissed at everyone involved in that little problem, it turns out it was a design flaw in the transfer switch. Because it happened during the day we ended up taking more of an outage for replacement of the switch then we did from the incident but it just proves that even a well designed system can have problems. That datacenter was small enough to only have single source power, my current datacenter has dual feed including dual generator and fully redundant cooling so a single transfer switch malfunction wouldn't take it down but you have to work within the parameters set by budget and need.

Re:Which only shows

[Eivind]

Bandwith is becoming less of an issue, though installation-costs still are. The reason is that a single single-mode fibre can carry literally hundreds of Gbps the current state of the art is around 40 Gbps for a single wavelength, and then there's wavelength-multiplexing.

In short, a single fibre will provide all the bandwith many datacentres need. Offcourse they'll want 3 independent ones for redundancy, but that's a different issue.

In short, installing a single low-latency 1megabit link somewhere far from civilization probably costs 90% of the cost of installing 100Gbps to the same location.

Re:Which only shows

[lb746]

I would like to see this idea taken to the next level and have the systems water cooled. Moving cold water from outside to inside would be a great idea. Just as long as it doesn't freeze in the lines.

Re:Which only shows

[NickCatal]

Hell no, the hosting company I do some work for left after that.

Equinix makes your CEO happy when he/she tours it, other than that it is just glitz.

They charge a fortune. New facility in the same building hasn't had a problem.

Re:Which only shows

[PitaBred]

Now imagine a machine with 2-3.06GHz P4 style Xeons, 3 10KRPM SCSI drives, 5GB of RAM and a GeForce 6600GT... ;) My computer room has the window open in the winter. Or at least a fan in the doorway to help redistribute the heat.

Re:Which only shows

[NickCatal]

I meant to remove the "No big deal" part of my post... It was part of a sentence that I thought I removed.

Re:Which only shows

[ByOhTek]

number of sticks is probably more important...

8x128MB is 1GB, but probably produces more heat than 2x2GB+1x1GB...

Re:Which only shows

[mpoulton]

Aren't most (all?) air conditioning systems ducted to the outside world?

Interestingly not. All modern HVAC systems have some outside air flow, which is required by building codes to keep the indoor air "fresh", but only a small fraction of the air is actually exchanged. For the most part, these systems are mostly closed. In some special applications like data centers, operating rooms, and cleanrooms, the system may actually be completely closed during normal operation.

Re:Which only shows

[LiquidCoooled]

The system is closed however the heat exchanger unit operates to the (usually) cooler air of the outside world.

Re:Which only shows

[Azarael]

That's true, but I would say that it's a vendor's responsibility to ensure that. No one can fully cluster your app for you.

Re:Which only shows

[initdeep]

actually a geothermal heat pump (or several). properly sized of course, could help efficiency a large amount. However, you better have a helluva big field to do drops in, or a good cold river flowing by to transfer from....

Re:Which only shows

[Kalriath]

Ok, I specifically said UPS power, as in it takes time to spinup the generators and switching from one source to the other does not always go perfectly in the real world. One factor is minimum cycle time on the compressors. The 3 minute time frame was from TFA which says that at a density of 5KVA per cabinet thermal shutdown can happen in 3 minutes due to thermal load. How long can it really take to bring your generators online? I ask because I work at a hospital, and we have three forms of power provisioned across the facility. Ordinary Power, in the case of a utility failure, is simply gone. This is used for unimportant things (the TVs in the lobby, most computers, main lighting, tea rooms, etc). Essential Power, which in the instance of utility failure is only out long enough for our generators to start, and feed the circuit. And Uninterrupted Power, which in case of utility failure is fed by battery backup until the generators start.

When we actually need to flick to generators, it takes them 30 seconds to start and take control of the power feed. Although our servers are on uninterrupted power (same as the Air Con) they could survive long enough for essential feed to take over if the Air Con wasn't on uninterrupted.

Is there any reason datacentres have a slacker design than us?

Re:Which only shows

[rkanodia]

That's kind of a tautology.

Re:Which only shows

[Maxo-Texas]

A usual problem with failovers is that testing them causes downtime. So you do them less frequently to reduce downtime- which increases the downtime for the failure (since now it is a special procedure instead of something you do every week) and so you do it less frequently until finally, you don't fail over any more except during real emergencies so it is even less reliable and subject to random slowness.

Re:Which only shows

[Richthofen80]

The coolers were on backup power. They just broke during the sudden transfer of load from utility to backup.
Here is the update from myrackspace.com, their user portal (I do independent software development and consulting, and I put my clients on rackspace whenever possible).

Nov. 12th 8:30PM CST -- In a completely unrelated incident to this weekend's power problems in DFW, a traffic accident caused damage to a power transformer which provides utility power to our DFW data center. Here is the current sequence of events:

        * At approximately 6:00 p.m. CST utility power was lost to the DFW data center
        * Power automatically switched over to backup generators without disrupting service for any customers
        * When generator power was established two chillers within the data center failed to start back up
        * Utility power was re-established through a secondary utility source
        * As a result of temporary data center temperature increases, we proactively shutdown a number of customer servers to protect them from overheating

At this point, the chillers are back up and running and we are operating on generator power throughout the data center. We have contractors on site to repair the damage and will be in contact with all customers who have been affected by this outage. We apologize for any disruption to your business operations and will work diligently to restore your service.

Re:Which only shows

[Critical Facilities]

Although our servers are on uninterrupted power (same as the Air Con)

I guarantee your HVAC systems are NOT on UPS power. If by some massive failure during construction and commissioning they were and it was missed, I'd recommend firing your entire engineering department and any development contractors involved with building and maintaining your facility. There is no reason to put HVAC systems (chillers, pumps, air handlers, CRACs) on UPS as they can all manage just fine with losing their power and restarting once power is restored (either from utility or generator). To subject your UPS system(s) to the massive inrush current that would occur when various HVAC component loads are thrust on it would be....well, stupid at best.

Your power systems sound pretty consistent with what is in most Data Centers (the "Essential Power" is often referred to as Emergency Power in Data Center environments). 30 seconds is a pretty good turnaround time for generators to start up, although 15 seconds is better (and very attainable).

So to answer your question, no, Data Centers do not have a "slacker" design than hospitals. They are actually quite similar in their requirements in terms of HVAC and of course power.

Re:Which only shows

[Kalriath]

I suppose it's worth pointing out that due to the lesser size of our facilities, our AC systems aren't as large as your average datacentre - the backup power (which I get the feeling "isn't your average UPS") might be able to sustain them (pretty sure the Air Con here isn't high voltage). Now, that said, it is possible that they do fail for the 30 seconds it takes (i.e. they're possibly on Emergency Power) for the generators to take over, I've never stood in the server room during an outage. Seeing that the door security system goes down during outages, I don't think the server room is accessible anyway (without the authorisation system up, the mag-locked doors wont open).

A good chunk of our servers do shut down during outages anyway (non-essential ones. QA, Training, UAT environments) so the AC load decreases hugely anyway.

Re:Which only shows

[shoemakc]

Actually, hospitals are way more complex then most datacenters.

Here in the US, hospitals are required by to have not one, but three "branches" of emergency power, all fed from different transfer switches, with separate distribution...in addition to any normal (non-emergency) power. They are:

"Life Safety" - Anything essential for detecting danger and getting people out of the building (fire alarm, exit lights..etc)
"Critical" - Anything essential for maintaining life during an outage (portable life support equipment...etc)
"Equipment" - Any larger mechanical equipment required for maintaining hospital operations during an outage

Also, to meet inspections, hospital generators are required to start up in a maximum of 10 seconds (though they often take far less time...as little as 3 seconds). Of course this startup speed required additional equipment like engine block heaters, oil heaters....etc to keep the generator as close to it's operating state as possible. And course none of this is even touches on UPS power....which an increasing amount of medical equipment now requires.

I guess my point is that while you might be very proud of your data center setup...keep in mind that Hospitals are about the most complex buildings you can find; they're required to be.

-Chris

Re:Which only shows

[scheme]

He's not talking about the local bandwidth, within the supercomputer. He's talking about the bandwidth in and out of the supercomputer, to the rest of the internet. Most traditional supercomputering applications involve loading a large dataset, but just once, and then waiting a long time, and downloading a smaller result. Much less bandwidth, in terms of ISP bandwidth to the rest of the world, than say hosting youtube.

At least with particle physics those 1-5 petabytes of data need to be sent out to sites to be processed initially as well as later when someone wants to analyze events. Most institutions don't have a few petabytes of storage handy so if a local user needs to run an analysis that involves events x,y,z, the data concerning those events get pulled and transferred.

I'm sure similar things happen with the bioinformatics people so yes the data gets shuttled around pretty frequently.

Re:Which only shows

[drsmithy]

If you want 100% uptime, it's important to have back up power for the cooling as well as the server systems themselves.

If you want 100% uptime, it's important not to have all your servers in the same physical location.

Re:Which only shows

[Critical Facilities]

Actually, hospitals are way more complex then most datacenters.

Well, it's clear by that statement that you have no idea of the infrastructure of a Data Center.

hospitals are required by to have not one, but three "branches" of emergency power

Well, Data Centers have this same distribution system as well. However, it's a little more complicated than you're explaining it. For example, an "Equipment" feed (as you call it) that is comprised of "Any larger mechanical equipment required for maintaining hospital operations during an outage" sounds pretty vague. In truth, the 3 main types are Utility, Emergency, and UPS. Utility has no backup, Emergency goes down with utility, but comes back up with the generators, and UPS never goes down ever.

hospital generators are required to start up in a maximum of 10 seconds (though they often take far less time...as little as 3 seconds

"Starting" and closing to the Buss are 2 very different things. If you believe that large generators are starting and closing to the buss at full voltage and balanced frequencies in 3 seconds, I have a bridge that you may be interested in purchasing. To give you some perspective, our 2 generators for our Data Center (2 Megawatts each) start and close to the buss (and are assume the building load) in 15 seconds. We, of course, circulate the heated jacket water to keep the oil, cylinders, etc warm and ready as you described.

I guess my point is that while you might be very proud of your data center setup...keep in mind that Hospitals are about the most complex buildings you can find

I am proud of our Data Center (and the others I've had the honor of running). I would not argue that hospitals are complex facilities, but to argue that they're more complicated than Data Centers is both short sighted, underinformed, and well, petty. There are many components that you're probably unaware of and layers of redundancy that are invisible to those who do not work in the "back of house" Critical Environments. To reiterate, I'm not saying that hospitals aren't complex nor am I saying that they do not have Critical Environments within them. I'm simply saying that you may have a perception of what a Data Center is that is not necessarily consistent with what is actually the case.

Re:Which only shows

[shoemakc]

Well, it's clear by that statement that you have no idea of the infrastructure of a Data Center.

Believe it or not, I've designed both, and while I certainly don't claim to be an expert on all the IT equipment, I've got a pretty good idea of the electrical systems that go into them.

My description of the emergency branches was intentionally vague because their full definitions comprise some dozens of pages in NFPA 99. I assumed most people wouldn't care about that level of detail :-)

Anyway, my point was that while a typical data center has 3 types of power available (Normal, Emergency and UPS), a typical hospital usually has at least 5:

Normal
Emergency (Life Safety)
Emergency (Critical)
Emergency (Equipment)
Emergency (UPS)

These generally include separate panels, feeders, automatic transfer switches...etc, so I still stand by my claim that hospitals have the more complex electrical system. Also consider that hospitals now contain increasingly critical data center facilities. Of course I will concede that the UPS topology of a large data center is generally far more complex then a hospitals....but again, that's just one part of the puzzle.

"Starting" and closing to the Buss are 2 very different things. If you believe that large generators are starting and closing to the buss at full voltage and balanced frequencies in 3 seconds, I have a bridge that you may be interested in purchasing. To give you some perspective, our 2 generators for our Data Center (2 Megawatts each) start and close to the buss (and are assume the building load) in 15 seconds. We, of course, circulate the heated jacket water to keep the oil, cylinders, etc warm and ready as you described.

I'll take that bridge. The reason your generators take 15 seconds to start is that they comprise a Level 2 system (as defined in NFPA 110), and not the Level 1 system that hospitals require. Level 1 includes a whole bunch of additional requirements (ie...expense) that are simply not required where the outage will not potentially risk human life, ie, datacenters. Now i'm not sure about all the modifications that manufacturers must make to their gen sets to meet these requirements, but I can assure you that that 10 second start (which includes startup, sync and bus connection) is required by code. Also, I've been there at the monthly test that hospitals are required to perform and yep...they really are that quick.

Now again...it's not that your generators are bad...it's just that theirs no reason for a company to spend the extra cash on that sort of system when a longer startup time will do; typically the UPS is sized for 15 minutes of runtime and the HVAC equipment can go down for a few minutes without the room overheating.

There are many components that you're probably unaware of and layers of redundancy that are invisible to those who do not work in the "back of house" Critical Environments. To reiterate, I'm not saying that hospitals aren't complex nor am I saying that they do not have Critical Environments within them. I'm simply saying that you may have a perception of what a Data Center is that is not necessarily consistent with what is actually the case.

Similarly, I'm not claiming that hospitals are more complex overall systems....just that their electrical distribution systems typically are.

-Chris

Re:Which only shows

[slashuzer]

Just as long as it doesn't freeze in the lines. That should be easy enough to do. Just have a 'start mode' which unclogs and gets the circulation going with supplied heat, and during operation the hot water itself can be used to prevent freezing.

Better still do away with water all together and use a fluid more suitable to such tasks, just make it closed cycle so it all acts like a giant radiator.

Re:Which only shows

[budgenator]

you'd trade AC bills for heating bills, computers won't run at -70F

Re:Which only shows

[budgenator]

If you are real serious about uptime then you'd have enough heatsink to cycle from primary utility, secondary utility and backup power twice. We just build a new jail and after it's first blackout, the primary contractor said to the county "If you wanted the toilets to flush during a blackout, you should have spec'ed it." These cretins though that the admin areas and the electric toilets would be on the same backup generators as the secure jail areas!

Re:Which only shows

[Critical Facilities]

Believe it or not, I've designed both,
I'm having a hard time believing that, at least with regard to the electrical infrastructure. For one thing, there are some inconsistencies emerging in your responses (i.e. going from claiming that there are 3 "branches" of electrical service at a hospital to claiming there are 5, or even dozens without really elaborating on the significance) We too have various portions of the Emergency system that are dedicated to Life Safety or Equipment or Critical/UPS(for example), but I don't see the significance of calling one portion of your Emergency Service Critical, one UPS, one Equipment, etc. I'm not saying that you don't have these differentiations, just that with respect to this conversation, I don't see how it applies, given that each of these systems is either a) provided Utility only power b) backed up by generator(s) or c) powered by UPS.

In addition, NFPA 110 does differentiate between Level 1 and Level 2 systems, however, it's not the Level but the Type that determines the timing restrictions (plus, you've moved back from your 3 second claim to a more reasonable 10 seconds anyway, so no bridge for you ;-) ). Obviously, the risk for loss of life is far greater at a hospital, but you must keep in mind that in Data Centers, losing the Critical Load costs millions of dollars so the "expenses" you refer to in generator architecture are more than justified financially. Also, if you've designed Data Centers, you would know that (obviously) each of the various electrical feeds are also fed from different "panels, feeders, transfer switches" as well as having Automatic Throw-over switches, Static Switches, Smart Switches, redundant PDUs, Control Cabinets, Harmonics Mitigating Transformers, etc....I could go on and on.

I guess what I'm saying is, while there are different applications for both of our respective Critical Environments, there's no evidence that a hospital is more "complex" (a term that's hard to quantify anyway). I'm not really interesting in a contest (i.e. "my generator can beat up your generator"), particularly with what sounds like a "brother in arms" in the sense that it seems you also work in these types of environments. I think it's more interesting to clear up misconceptions where it's productive as in the original subject of this thread, which was essentially whether or not HVAC systems would be powered by UPS, which I'm sure you'd agree they are NOT.

How to estimate the cooling needs?

[Dynedain]

Actually this brings up an interesting point of discussion for me at least. Our office is doing a remodel and I'm specifying a small server room (finally!) and the contractors are asking what AC unit(s) we need. Is there a general rule for figuring out how many BTUs of cooling you need for a given wattage of power supplies?

Re:How to estimate the cooling needs?

[CaptainPatent]

Is there a general rule for figuring out how many BTUs of cooling you need for a given wattage of power supplies? I actually found a good article about this earlier on and it helped me purchase a window unit for a closet turned server-room. Hope that helps out a bit.

Re:How to estimate the cooling needs?

[Critical Facilities]

Try this.

Re:How to estimate the cooling needs?

[trolltalk.com]

Believe it or not, but in one of those "life coincidences", pi is a safe approximation. Take the number of watts your equipment, lighting, etc., use, multiply by pi, and that's the # of btus of cooling. Don't forget to include 100 watts per person for body heat.

It'll be 90F degrees outside, and you'll be a cool 66F.

Re:How to estimate the cooling needs?

[fifedrum]

It's not really this simple, but a decent back of the napkin is to take the amperage the voltage and multiply, then multiply again to get btu/hour, divide to get tons.

20A x 110V = 2200 VA which doesn't directly translate to Watts, as someone will surely correct me, but for cooling purposes it's not a bad rule of thumb to directly translate the VA to Watts because you'll be including a built-in overhead into which you will surely grow your server space. Then go from Watts to BTU/hour.

2200 Watts x 3.412 BTU/Hour Watt = 7506 BTU/hr

12000 BTU/hr = 1 ton. Do that calculation for all possible hosts in your space, round up. Then purchase an additional, but portable, cooler for the space. Use that cooler for emergencies, like chilling beer, and if the main chillers break, you'll have nice cold beer to drink while the HVAC guys fix the big units and you wait for your less-essential machines to come up.

Most people will do the caclulaton and find their datacenter cooling systems are woefully under sized, running 100% whenever the outside air temperature is above 50F...

Re:How to estimate the cooling needs?

[Smidge204]

1 Watt = 3.41 BTU/hr

So if your server/network setup has 1000 watts of power supply capacity, I would recommend no less than 3410 BTU/Hr of cooling capacity, or 3 tons of cooling. This is given that power supplies don't usually run at their peak rated capacity, so it's slightly more than you technically need if there were no other sources of heat. Add in 10-25% for additional heat gains. Add 250 watts on top of that for each person that may work in that room more than an hour at a time.

Final formula I used on the job:

[1.2 * (Max rated wattage for all equipment)] + [250 * (Max number of people)] * 3.41 = BTU/Hr cooling

(1200 BTU/Hr = 1 ton of cooling, as some cooling equipment is rated in tons)

Of course, consult your local P.E. or cooling equipment manufacturer since there may be unique things about your situation.
=Smidge=

Re:How to estimate the cooling needs?

[xaxa]

WTF? I thought 'ton' was a mass/weight, but you have BTU/hr which is energy/time, which is power.

Crazy Americans :-p

Re:How to estimate the cooling needs?

[MrLogic17]

I've always heard taht 100 watt/person, but have never seen real data to back that up.

According to
http://www.swarthmore.edu/NatSci/es/energy/bodyheat.html
A person puts out 54 watts. Or at least 1 standard professor unit does.

Re:How to estimate the cooling needs?

[DamnStupidElf]

Don't forget that nerds and geeks often have much more surface area than the standard professor, leading to higher heat loss into the cooler server room.

Re:How to estimate the cooling needs?

[Bandman]

I hate to pick, but I think you'll find rotund people actually have a lower surface to mass ratio than thin people.

Re:How to estimate the cooling needs?

[JUSTONEMORELATTE]

Believe it or not, but in one of those "life coincidences", pi is a safe approximation. Take the number of watts your equipment, lighting, etc., use, multiply by pi, and that's the # of btus of cooling. Don't forget to include 100 watts per person for body heat.

It'll be 90F degrees outside, and you'll be a cool 66F.

And if that doesn't work, you can always tell your VP that you were taking your numbers from some guy named TrollTalk on ./
I'm sure he'll understand.

Re:How to estimate the cooling needs?

[FuzzyDaddy]

To calculate the average IT worker heat dissipation, take the daily number of Calories in the diet, converted to joules, divided by the number of seconds in a day.

This assumes no actual work is being done.

Re:How to estimate the cooling needs?

[Anonymous Coward]

> Don't forget that nerds and geeks often have much more surface area than
> the standard professor, leading to higher heat loss into the cooler server room.

Not to even think about the actual situation during power outtakes:

When power is lost, all the geeks are going to be really excited and sweating (and heating) very much more than normally!

Re:How to estimate the cooling needs?

[Nf1nk]

Personal energy output is a function of a number of variables, but the most important, are the ambient temperature and the movement of air through the room. The 100 watts per person is a conservative estimate based on a roughly 75 F room.

The Prof in a box experiment has a large issue that contributes to error. He is breathing with a tube, the heat exchange in your lungs is a convection exchange and has too large a magnitude to ignore. If you have doubts about how much heat flows out through breathing next time you are cold in bed pull the covers up over your head and breath under the covers. You will find that the bed gets nice and warm in a very short time.

Re:How to estimate the cooling needs?

[Anonymous Coward]

Well, specifically, a ton, in HVAC terms, is the amount of energy one ton of ice can absorb. i.e. when refrigeration replaced ice, if you had one ton of ice delivered per day, a one ton cooler would fit the bill.

Re:How to estimate the cooling needs?

[R2.0]

The general rule I always follow is HIRE AN ENGINEER! Ferchrissake, there are people who do these calculations for a living, and have insurance in case the screw up. You want to trust your data center to advice from slashdot and a back-of-the-envelope calculation?

Sheesh - what's the name of your company so I can sell short?

Re:How to estimate the cooling needs?

[xaxa]

Ah, thanks -- I'm glad there's some reasoning behind it!

Re:How to estimate the cooling needs?

[afidel]

Your numbers are off by a factor of 10 because one ton = 12,000 BTU/hr! I guess you have WAY too much cooling capacity, which isn't good because you will be constantly cycling your cooling which is way less power efficient than running a properly sized unit more or less continuously.

Re:How to estimate the cooling needs?

[zippthorne]

Recommended diet is typically between 1500 kcal and 2500 kcal depending on body type and other factors. It is, I believe, a measure of how much energy you would be able to extract from the food, rather than what you'd be able to obtain by burning it directly.

Anyway, if you use an average of 2000 kcal, whether that goes into heating or moving around, a control volume around yourself will experience the same thing: 2000 kcal of waste heat generated over the course of a day. Everything turns into waste heat, eventually. The chemical energy you are unable to extract from the food is irrelevant: it will be passed and remains available for something to extract in your refuse.

Further assuming that your usage of energy is mostly constant (i.e. that your physical exertions are << the things you keep doing to stay alive (generate heat, pump blood, etc)) you would assume that a person would, on average, be a 100 W heat source.

So, what does this mean for the professor's experiment? Well, obviously practical experience trumps theory, so it has to be reconciled with the 2000 kcal estimate on diet. Is the professor eating enough? He looks healthy. Perhaps he's not moving enough. Is the air-displaced relevant? I'd think correcting for it (they cleverly used a capacitance method) would reduce he estimates of the professor's output. Perhaps estimates of energy conversion are incorrect and need to be corrected.

The only glaring flaw I can see is that the breathing tube does not appear to have any temperature probes or flow meter. This is important because, due to the huge area of the lungs, every breath will reach equilibrium temperature inside the professor, so there is the question of how much is lost in that manner.

I've always heard ~50W, as well. Though the food-calc suggests otherwise. 50-100W isn't really a bad estimate though, since you'd only be using it to make order of magnitude calculations, anyway.

Re:How to estimate the cooling needs?

[rMuD]

Cooling is only part of the equation, humidity control and a Air Handler that keeps the air moving 100% of the time is just as important. Putting a wall unit in that turns off and on every 2-3 minutes isn't going to last long. Depending on the region you live in your humidity is going to vary from 0% to 100% depending on the seasons.

You need a System that is designed for a computer room, worst case, spec a system that can maintain your humidity at 50%, and the blower running 100% of the time re-circulating, temperture change is more problematic than Heat alone. A system that goes up and down in temp 2-3 degrees every 5 minutes is worse than keeping your servers in a 90F room.

I am also assuming your not doing raised floor, and most common servers now are front/rear cooling vs bottom/top so make sure they design the air flow in the room accordingly.

Re:How to estimate the cooling needs?

[zerocool^]

Chiming in with this: http://www.dell.com/html/us/products/rack_advisor/index.html.

Dell-centric, but Dell is what we use here.

It'll tell you how much power / cooling / rackspace / etc you need.

Re:How to estimate the cooling needs?

[TheHawke]

You always fudge and oversize your cooling systems to accommodate any further expansion of your data center. It's better to have excess (within reason) than to get shortchanged if it drops into the bucket.

I know of one central plant at a university that has two empty holes in their 6 fan condenser row to install more fans if they are needed. Plus a 100K gallon chill water tower to provide reserve and surge capacity.

I'll bet they did not have any reserve on hand when they went to candle power.

Re:How to estimate the cooling needs?

[zerocool^]

Actually, the more I look at the other responses to your thread that have been modded up, they really seem to be more abstract power-to-heat conversion physics-type things. That link from dell above is really what you want; it converts from "servers" into "Amps needed" and "Tonne rating on Air Conditioning unit". You can approximate based on the server models; just look them up on Dell's website to see what the configuration that comes with each server is, find one that approximates your setup, and roll with it.

~Wx.

Re:How to estimate the cooling needs?

[R2.0]

The general rule that I follow in such a situation is HIRE A PROFESSIONAL! There are people who do this stuff all the time, and actually have insurance in case they screw up their calcs - they're called HVAC engineers. Use one.

And please let me know the name of your company so I can sell it's stock short - what the fuck are they doing having someone "specifying" a server room who can't figure the loads?

Re:How to estimate the cooling needs?

[trolltalk.com]

Think for 2 secs ... each kw of electricity eventually gets converted to heat. Resistive heating generates ~ 3,400 btus per kilowatt, so multiplying electrical consumption by pi gives you a decent cooling capacity. Add an extra 10% and you're good to go (you *DO* remember to add in a fudge factor of between 10 and 20% for "future expansion", right?)

Re:How to estimate the cooling needs?

[Smidge204]

You're absolutely right, and I'm embarassed that I made such a mistake.

Please ignore everything related to "tons" in the original post. The rest of it is fine, though. :p

On the flip side, some people gladly abuse their system cooling equipment rather than maintain the equipment itself, so depending on the specs of the unit an abnormally high degree of oversizing is not as bad as, say, a typical office or residential application. A good, commercial air handler with DX cooling can modulate quite a bit, and I've known more than one sysadmin that would rather turn the thermostat down to 50 degrees than clean the air filters on his server cabinets. (Actually had to have a manufacturer's engineering rep give the okay that the unit wouldn't die when trying to do that. It's an air handler not a refrigeration unit!)

But yeah, factor-of-10 is inexcusable. My bad.
=Smidge=

Re:How to estimate the cooling needs?

[Sloppy]

Ok, take your constants from someone named Sloppy. My quick guess is that it's 314.

Re:How to estimate the cooling needs?

[Dynedain]

• Small Privately held company (less than 100 employees)
• IT or technology-related issues are not the core of the business
• Doing mostly architecture, we have a lot of experience w/ HVAC, but our HVAC consultants can't answer because they don't do server rooms, and the contractors need an answer.
• A consultant to pick the right HVAC unit will cost as much as the HVAC unit itself
• We don't need someone to tell us if we need 2.5million BTUs vs. 4.5million BTUs, we need just a general figure of 30,000 BTUs vs. 300,000 BTUs.
• I said this was a small server room... probably no more than 3-4 racks of equipment will ever be in it. This isn't a datacenter and we don't run client services off of them, it's the machines needed to run our company during business hours.

Sheesh, get a grip and understand scope. I need to stick at least 25, and up to 60 machines in a closed room and want to know how much AC I need. I don't need a high-end HVAC consultant experienced in data centers for that, just like I wouldn't call up SAP just because I need to track timecards.

Re:How to estimate the cooling needs?

[R2.0]

"but our HVAC consultants can't answer because they don't do server rooms, "

and

"We don't need someone to tell us if we need 2.5million BTUs vs. 4.5million BTUs, we need just a general figure of 30,000 BTUs vs. 300,000 BTUs."

Apparently, your consultants think you DO need someone to tell you that, because it is outside their area of expertise.

And it may not be the core of your business, but that doesn't mean it's not *critical* to your business. See what happens to your deadlines because your file servers are down because the AC units were sized on the back of an envelope.

Your contractors need the RIGHT answer; are you sure it's the one you get from Slashdot?

Re:How to estimate the cooling needs?

[DamnStupidElf]

Heat transfer to the same environment depends directly on surface area, assuming bodies of the same temperature. Until hypothermia sets in, skin temperature is probably going to be pretty stable across individuals.

Re:How to estimate the cooling needs?

[Bandman]

Ah, I see what you mean. So as they have more mass anyway, they're more likely to produce more btu / inch^2

Re:How to estimate the cooling needs?

[fifedrum]

you are right, it is excess.

two chilling units each capable of putting out 100% of your current cooling needs is redundancy.

But when your manager decides to add racks full of blade servers and hard drives by the hundreds, this redundancy becomes "excess capacity". Much better to have excess capacity now (paid for up front) than to have to refit your data center with bigger ducting, higher power feeds, bigger chillers, bigger fans, bigger heat exchangers when you find "excess capacity" has changed to "cooling deficit".

for a guy with a closet full of PC class servers, yeah, get a cheap wall mounted AC unit and be done with it. For a guy trying to cool a whole raised floor full of machines, 200% capacity is just about right.

Underground thermal energy

[logik3x]

http://www.doc.ic.ac.uk/~matti/ise2grp/energystorage_report/node5.html

You could do like my previous Director of IT did..

[apparently]

When systems start shutting down because the on-board temperature alarms trip, just disable the alarms.

Man, I wish I was making that up.

And the answer is: Liquid Nitrogen

[Bombula]

Liquid nitrogen is the cooling answer, for sure. Then you're not dependent upon power of any kind at all. The nitrogen dissipates as it warms, just like how a pool stays cool on a hot day by 'sweating' through evaportation, and you just top up the tanks when you run low. It's cheap and it's simple. That's why critical cold storage applications like those in the biomedical industry don't use 'chillers' or refrigerators or anything like that. If you really want to put something on ice and keep it cold, you use liquid nitrogen.

Re:And the answer is: Liquid Nitrogen

[Radon360]

Seems that having an industrial-sized tank of LN2 outside the building for such a purpose might make sense as a rather inexpensive emergency backup cooling system. Diesel generators keep the server farm online while cool N2 gas (after evaporation) keeps the server room cool. Just keep the ventilation system balanced properly that you don't displace the oxygen in the rest of the building, too.

And that brings up one caveat: You wouldn't have access to the areas cooled without supplied air when such a system is in operation (that's why it would be an emergency backup). Though, you could get back in fairly quickly (matter of minutes) once normal power is restored and the room is ventilated.

Re:And the answer is: Liquid Nitrogen

[everphilski]

Liquid oxygen?

The boiloff is a little bit worse but the stuff is almost as cheap as dirt. A mix of LOX and NOX would be breathable and not risk explosion.

Re:And the answer is: Liquid Nitrogen

[The -e**(i*pi)]

You can stick your whole hand in it for about 1-2 seconds 4 or more seconds may make it turn white colored tho, but warm water fixes that. And I speak from personal experience with a dewar flask of liquid N2, and be careful because those giant containers can freeze open and spray all the liquid nitrogen out, true story and you should have seen the wrench we needed to shut off the valve. I never got to play with the liquid helium because its too expensive.

Re:And the answer is: Liquid Nitrogen

[DerekLyons]

Liquid nitrogen is the cooling answer, for sure. Then you're not dependent upon power of any kind at all.

Except of course the power needed to create the LN2.
 
 

That's why critical cold storage applications like those in the biomedical industry don't use 'chillers' or refrigerators or anything like that. If you really want to put something on ice and keep it cold, you use liquid nitrogen.

As above - how do you think they prevent the LN2 from evaporating? The LN2 is a buffer against loss of power, but typically they have a pretty serious cryocooler to keep the LN2 there when they do have power.

Re:And the answer is: Liquid Nitrogen

[skeevy]

Liquid nitrogen is the cooling answer, for sure. Then you're not dependent upon power of any kind at all.

But you are dependent on a near-constant convoy of liquid nitrogen trucks. Some quick math (lots of rounding and approximations here):

• One gram of nitrogen can remove about 400 Joules of energy, going from about 77k (LN temp) to 297k (room temp). That includes boiling.
• a 5kw rack produces 5000 Joules of energy per second.
• so you need 12 grams of LN per second to keep it cool. That's assuming 100% efficient heat transfer.
• LN has a density of 0.8 g/cc, so you that's 15ccs per second.
• so you need a liter of LN every 66 seconds.

That's One Gallon every four minutes, or 360 gallons per day, per rack. Fifteen racks is 5400 gallons, more than one tank truck per day.

That's a lot of LN. It's great for things that don't produce heat, like tissues or organs, or okay for emergency or burst cooling, but not so great for absorbing steady-state heat production.

Re:And the answer is: Liquid Nitrogen

[EmagGeek]

I would argue against this as viable.

The heat of vaporization of Liquid N2 is 199kJ/kg, meaning it takes 200,000 Joules to evaporate 1 kg of liquid nitrogen. If a cabinet is dissipating 2kW of power (which is 2kJ/s), it will take just 100 seconds to evaporate 1kg of liquid N2. That's 36kg/h of N2, per cabinet. If a datacenter has 100 cabinets, that's 3600kg of liquid N2 per hour to keep it cool. At 0.808 kg/m^3, that translates to about 4455 cubic meters of liquid nitrogen. That's a volume 20 meters on a side and 11 meters tall, which is about the size of a small apartment building.

Not very practical, if you ask me.

Re:And the answer is: Liquid Nitrogen

[UnknowingFool]

It's not that simple. If it were then others would have already implemented it. Liquid nitrogen has issues with it.

The biomedical industry has used LN2 because they have different needs. They use it for cold storage (keeping things very cold). For computers, you need cooling (heat transfer). Also the footprint of their needs is small. The industry stores small things like vials, jars, etc. The computer industry needs to chill an entire rack. The average tank for the biomedical industry is about the size of a refrigeration and everything is sealed. To chill a rack, you need access points for wires and cables and building a tank to fit a rack is very expensive.

Even if it were possible to develop the right equipment for using LN2 in a computer rack there are issues with the production of liquid nitrogen. While cheaper than other gasoline, making liquid nitrogen presents some challenges as to creating enough supply for all that may be required. Then there are logistics to getting it to the customers.

Then there are challenges to using it. Getting enough LN2 to cool an entire data center would require massive storage tanks which have to be located away from the building for safety reasons. Then a distribution system has to be built to get the LN2 to the computers from the tanks. Then the whole system has to be monitored for safety and efficiency. While N2 is not toxic to humans in itself, the ratio of N2 to O2 in air must be maintained for safety.

Re:And the answer is: Liquid Nitrogen

[PPH]

Not really. You still need to make the liquid nitrogen. Or buy it. The infrastructure needed to handle it isn't cheap (cryogenic tanks, etc.). And an LN2 leak inside a confined space is dangerous. It will displace breathable air and kill all the occupants.

Take a look at the specific heat of LN2 and compare it to the specific heat of water, chilled to just above freezing. Taking into consideration the much simpler (and safer) systems needed to handle water, you are better off chilling a large insulated tank of water and using that as an emergency heat sink. One can also use this technology to shift the energy consumption of the cooling system to off peak hours.

The admins can also store a couple of kegs in the chilled water tanks for killing time between patch Tuesdays.

Re:And the answer is: Liquid Nitrogen

[MyNymWasTaken]

297 K = ~24 C = ~75 F

Re:And the answer is: Liquid Nitrogen

[rhizome]

Forgive me, but isn't 297k really cold for room temp?

297 Kelvin = 74.93 deg F

Re:And the answer is: Liquid Nitrogen

[MenTaLguY]

Even oxygen levels elevated to as little as 23% oxygen can lead to a violent increase in the flammability of materials like cloth and hair. Controlling gas concentrations so they remain at safe levels can be very tricky.

Setting aside evaporation, be careful not to get it on anything. LOX can easily saturate anything remotely porus and oxidisable, effectively turning it into an unstable explosive until the LOX evaporates... at LOX or LN temperatures, that can even become an issue with oxygen condensing from the air onto your equipment/insulation. Forget just avoiding the creation of sparks -- better be sure that the safety measures have been successful in eliminating all LOX-incompatible materials and be careful not to bump anything too hard!

And of course, even a tiny fire or explosion can easily lead to a rapid boiloff. Sudden boiloff can be an issue simply because of drastically increased pressure and still-cold temperature. Liquified gasses like LOX, NOX, etc. expand a LOT when they boil (about an 600-800x increase in volume, simply transitioning from a liquid to a gas), even while remaining dangerously cold. Imagine being in a closed room with a punctured dewar. Assuming you've escaped being hit by the dewar which has gone flying like a deflating balloon with reinforced-concrete-shattering force, you've potentially got ruptured eardrums and possibly internal injuries due to the abrupt pressure change which has also jammed the door. You fall to the floor from the pain of burns on your lower body from the ultracold gas which has quickly filled the lower part of the room -- which then starts to burn your face and lungs out too as you start breathing it.

Hopefully the facility you're in has proper emergency ventilation measures, adequate room size, properly constructed doors, and protective equipment to avoid this scenario, but you still don't want to be in the room if it happens if you can help it... Cryogenic gasses are seriously dangerous. Don't underestimate them or treat them lightly.

Re:And the answer is: Liquid Nitrogen

[Hoi Polloi]

Plus I assume you wouldn't want to use the LN2 as a coolant directly. Then you'd have condensation problems.

Re:And the answer is: Liquid Nitrogen

[zerocool^]

Um, you store LN in big thermos bottles (imagine the 5 gallon inverted watercooler jugs, but make them bigger so as to accommodate a vacuum layer, and made of metal). It keeps for a while.

Plus, to industrial applications, LN is about as expensive as milk.

Re:And the answer is: Liquid Nitrogen

[DerekLyons]

It keeps a while - but not forever. (Think a few days tops. It's a huge capital investment beyond that.)
 
It's cheap as milk when you order it in industrial quantities. (Which few activities other than well... big industries do.)

Re:And the answer is: Liquid Nitrogen

[aaarrrgggh]

We looked at using LIN for thermal storage and peak shaving. Benefit is that it is much more compact, but sizing an evaporator large enough for sustained gas flow was a problem. Even using heat recovery off an evaporator can be tricky without nearly 100% glycol.

The other challenge is thermal shock to the processors...

Re:And the answer is: Liquid Nitrogen

[danilo.moret]

No, liquid nitrogen is the cool answer.

Re:And the answer is: Liquid Nitrogen

[everphilski]

Hopefully the facility you're in has proper emergency ventilation measures, adequate room size, properly constructed doors, and protective equipment to avoid this scenario, but you still don't want to be in the room if it happens if you can help it... Cryogenic gasses are seriously dangerous. Don't underestimate them or treat them lightly.

I'm an Aerospace Engineer. I'm fairly familiar with the properties of cryogenic oxygen :)

The first thing you do, is you NEVER store a LOX dewar inside an enclosed area. You store it outside. When it vents, it vents into the atmosphere. You have a mixer outside which expands and mixes the LOX with the NOX. OUTSIDE. With proper feedback the appropriate breathing ratio can be attained. It's insulated so you don't lose that nice cool chill. Only when the mix is appropriate do you vent it into your server room. Otherwise, dump it outside ... it's nitrogen and oxygen, the primary constituents of the atmosphere.

LOX is dirt cheap. Pennies a pound by the tanker. There are many people who have small dewars at home that they own for medical purposes (migraines, cluster headaches, lung issues, etc.). There are even guys I know who make liquid rocket engines as a hobby, and have LOX dewars in their backyard. The handling of it is well known, to the point where oxygen is becoming a viable commercial industry (some have called it the next bottled water). With proper handling there is no reason to be intimidated.

Re:And the answer is: Liquid Nitrogen

[MenTaLguY]

I stand corrected.

Re:And the answer is: Liquid Nitrogen

[sjames]

Liquid nitrogen is only the answer if the servers need to be flash frozen and kept at -195.79 C.

The heat of fusion for water is greater than the heat of evaporation for nitrogen. In other words, the amount of heat taken up by boiling 1Kg of liquid nitrogen is only 2/3rd (approx) the amount taken up by melting 1Kg of ice. And the ice takes less space and requires no special handling.

As for cost, not counting the bulk storage tank (NOT cheap), it's about $8/Kg. Ice is less than half the price, can be made onsite when the power is on, and doesn't require a vacuum insulated storage tank.

Of course, to just keep things cool long enough to get the generators online and the A/C restarted, simple thermal mass in the form of adobe or even cinder blocks will do fine and present no maintenance issues.

New cooling strategy needed?

[MROD]

I've never understood why data centre designers haven't used a different cooling strategy to re-circulated cooled air. After all, for much of the temperate latitudes for much of the year the external ambient temperature is at or below that needed for the data centre so why not use conditioned external air to cool the equipment and then exhaust it (possibly with a heat exchanger to recover the heat for other uses such as geothermal storage and use in winter)? (Oh, and have the air-flow fans on the UPS.)

The advantage of this is that even in the worst case scenario where the chillers fail totally during mid-summer there is no run-away, closed loop, self re-enforcing heat cycle, the data centre temperature will rise but it would do so more slowly and the maximum equilibrium temperature will be far lower (and dependant upon the external ambient temperature).

In fact, as part of the design for the cluster room in our new building I've specified such a system, though due to the maximum size of the ducting space available we can only use this for half the heat load.

Re:New cooling strategy needed?

[Frosty Piss]

After all, for much of the temperate latitudes for much of the year the external ambient temperature is at or below that needed for the data centre so why not use conditioned external air to cool the equipment and then exhaust it...

The next major source of "global warming"...

Re:New cooling strategy needed?

[afidel]

The problem is humidity, a big part of what an AC system does is maintain humidity in an acceptable range. If you were going to try to do once through with outside air you'd spend MORE power during a significant percent of the year in most climates trying to either humidify or dehumidify the incoming air.

Re:New cooling strategy needed?

[silas_moeckel]

They do big data centers use glycol, when it's cool outside the compressors turn off and they just run the fans. It's a bit more up front but has savings in areas where it gets below 45 on a regular basis. Another option is large blocks of ice with coolant running through them to shift power consumption to the night and reduce the amount of energy required (cheaper to make ice at night) but only for smaller facilities they leave a reserve capacity of x hours and/or go with n+1 setups.

Re:New cooling strategy needed?

[The -e**(i*pi)]

air conditioners do the same thing, you didn't think they magically cool the air did you?

Re:New cooling strategy needed?

[cjanota]

Where do you think current AC units dump all the heat that they extract? What the GP is suggesting just cuts out the middle man (AC). The AC units produce quite a bit if heat themselves.

Re:New cooling strategy needed?

[R2.0]

Part of the problem is that it is a lot easier to move heat via liquid than air. the conventional design uses chillers mounted outside the space to cool a liquid medium/refrigerant, which is then pumped very efficiently to cooling coils in the space (modify for DX coils). the air inside the condityioned space makes a very short trip through the servers, across the room, over the coil, and back out again.

Under your scenario, the AIR is the working medium - it is cooled on the outside, and then moved inside via relatively inefficient fans. And it is a SHITLOAD of air - that means either high volumes (huge ductwork) or high velocity (how do you like working in a wind tunnel?).

Fans on UPS? Are you kidding? How big do you want your UPS to be? Fans suck a LOT of power, especially when you have them doing what you propose.

"as part of the design for the cluster room in our new building I've specified such a system"

You've specified? From your post, it's obvious you aren't an HVAC engineer, so what are your qualifications? Did you do an analysis to see what the real ROI is? Or is it just so obvious to you why years of HVAC design are totally wrong?

Re:New cooling strategy needed?

[swb]

You've specified? From your post, it's obvious you aren't an HVAC engineer, so what are your qualifications? Did you do an analysis to see what the real ROI is? Or is it just so obvious to you why years of HVAC design are totally wrong?

Years of a lot of designs in many fields have been total bullshit, often until someone who was an outsider to the field showed up and said "Why don't we try something different"? Groupthink and reliance on inherited wisdom often causes reasonable ideas to be rejected without serious consideration.

And I can't say I blame the OP for his idea -- I've often thought of it for my own home; if the outside temp sinks more than 2-3 degrees below the AC setpoint, the AC doesn't cycle and it gets stuffy in the house, so why not have a baffle that would intake outside air when the outside temp is below the setpoint, thus maintaining the setpoint without requiring running the compressor.

Obviously humidity is an issue, but it strikes me that dehumidification only shouldn't necessarily require running a coil with the 20-30 degree temperature gap that the AC coil requires, thus saving some energy. Even if intake air wasn't perfectly dehumidified on intake, the blower could cycle for a longer period allowing further dehumidification.

The OP's idea for cooling a data center makes even more sense for very cold climates, as it would require little or no dehumidification -- relative humidity in winter months is often 20% around here.

Re:New cooling strategy needed?

[R2.0]

What you and the OP are describing is called "free cooling", a long established principle in HVAC design. It is used in commercial and industrial buildings all the time. The reason that it is not used in residential all that much is that

1) until relatively recently, houses "breathed" quite well on their own due to loose construction. With tightening energy codes and the use of Tyvek and better windows, houses don't have a lot of air exchange through the boundaries, and problems ensue - "stuffiness", moisture, mold, "sick building". Residential construction hasn't thought this through yet - there are some builders who now refuse to use Tyvek due to ventilation (and liability) issues.

2) Controls become an order of magnitude more complicated. Most residential systems are "bang bang" systems - it's on or off based on 1 criteria. To introduce free cooling, you need outside air sensors, dampers, actuators, and a controller a lot more complex than a home t-stat. For most ersidential builders, that's a couple thousand in extra costs that can't be recouped in sale price - most owners just don't care, and when you are building 5000 of the same unit, "most owners" rule.

As for dehumidification, you have it backwards - dehumidification typically required a COLDER coil than necessary for cooling alone, and then you reheat the air. It is horribly inefficient, but sometimes necessary - with a "tight" building, you have to get the moisture out somehow, and supercooling the air inside just isn't a good idea (other than making for lots of erect nipples, that is)

Finally, what makes sense for one situation may not for another - a data center uses orders of magnitude more cooling than a house or common office building. Moving the amount of air necessary to provide that cooling gets really hard - the amount of energy a fan requires increase with the CUBE of the flow required. So to get twice the airflow you use 8x the power. It's the same with pumps, but because the heat capacity of water or glycol is so much greater than that of air, the effects are minimized.

Re:New cooling strategy needed?

[evilviper]

you'd spend MORE power during a significant percent of the year in most climates trying to either humidify or dehumidify the incoming air.

Dehumidifiers are generally regarded as requiring only 1/3rd as much power as air conditioners, and in some parts of the world, are quite popular.

Re:New cooling strategy needed?

[evilviper]

Another option is large blocks of ice with coolant running through them to shift power consumption to the night and reduce the amount of energy required (cheaper to make ice at night)

Ice systems are used because of peak/off-peak metering only. The energy they consume ends up being more than just running the AC directly, when it is needed, due to the inefficiency of running AC at that low of a temperature, losses due to imperfect insulation, etc.

If that wasn't the case, nobody would ever need AC. They'd just put insulation around their pools, and do heat exchange inside the building during the day, and with the outside air at night. In other words, thermal mass.

Re:New cooling strategy needed?

[evilviper]

it is not used in residential all that much

Not true. "Heat exchangers" as they are called, are quite commonly used in home HVAC to bring in a certain portion of outside air.

Controls become an order of magnitude more complicated.

Such residential heat exchangers don't require any complex additional controls to get a good portion of the benefits of "outside air boost" or whatever the marketing name is for commercial HVAC systems today.

dehumidification typically required a COLDER coil than necessary for cooling alone, and then you reheat the air. It is horribly inefficient

That's also untrue. The direct, free-flowing heat exchange between hot and cold coils allows dehumidifiers to be much more energy efficient, using typically around 1/3rd as much power for the same volume of air.

As to coil temperature, obviously any temperature will work, to varying degrees of effectiveness. You'll need to provide some numbers to back up your claim. General-purpose dehumidifiers are usually just slightly modified AC units.

So to get twice the airflow you use 8x the power.

No, what you do is install a second fan.

It's a patently ridiculous claim, anyhow. If you couldn't blow-in enough cool air from outside, then you couldn't move enough air to/from the AC units.

Re:New cooling strategy needed?

[R2.0]

Are you stupid? A heat exchanger (btw, not "very common" at all in residential) is the OPPOSITE of free cooling! In free cooling, COLD outside air is brought directly into the space, bypassing the cooling coils. Why? Because the air is cool already.

With a heat exchanger, you are bringing cool air in, and then HEATING IT UP with the waste heat from the exhaust air. Great for saving energy in a residence, when one wants to stay toasty warm - not so great in a data center or office building when there is still a cooling load in winter. So absolutely nothing you said has anything to do with free cooling. Heat exchangers are great for what they do, but free cooling isn't it.

"That's also untrue. The direct, free-flowing heat exchange between hot and cold coils allows dehumidifiers to be much more energy efficient, using typically around 1/3rd as much power for the same volume of air."

True - IF you are using a heat exchanger. But if one is not - lets say, in an office building on a cool spring morning - then you have a problem. You bring in nice 65F air, at 65-70% RH - it's wet. You don't heat it up through a HX, because you need the 65F air to maintain temp setpoint. But now you are dumping a lot of water into the space, and it doesn't *feel* cool. So, you run your cooling coil at, say 50F discharge temp. That is below dewpoint, and it pulls moisture out of the air. But now you are dumping 50F air into the space, so the space temp gets driven down, and you get the nipple effect. So what do you do? REHEAT the air to 65F. Which, BTW, is exactly what home humidifiers do - the discharge air is reheated to a temp greater than the intake air, reflecting the energy added by the electricity. TANSTAAFL.

You can throw a HX in that equation, but it certainly isn't a dumb device - the control logic needs to know when to open the air dampers and close them, so as not to interfere with free cooling.

"As to coil temperature, obviously any temperature will work, to varying degrees of effectiveness. You'll need to provide some numbers to back up your claim. General-purpose dehumidifiers are usually just slightly modified AC units"

Bullshit. The coil temperature MUST be less than the dew point of the air, by the very definition of "dew point". Practically, it needs to be substantially less for the dehumidification to really work. Often, that temp is less than desired for discharge air temp. See above example.

Call me when you've bought a psychrometric chart and a ductulator. There are plenty of design decisions to be made when designing an HVAC system, unfortunately including appeasing owners who think they are design geniuses.

Re:New cooling strategy needed?

[evilviper]

With a heat exchanger, you are bringing cool air in, and then HEATING IT UP with the waste heat from the exhaust air.

Yes you are, but heat exchangers aren't remotely perfect, so in cool weather you get a small portion of "free" cooling.

You don't heat it up through a HX, because you need the 65F air to maintain temp setpoint. But now you are dumping a lot of water into the space, and it doesn't *feel* cool.

This is a brutally stupid statement. Dehumidifiers run the outside air over the cool coils to dehumidify it, then over the hot coils to re-heat it. THAT is the heat exchange. You'd be an idiot to run it over the hot coils first, that's the exact opposite of what you need. Why you're confusing the simple operation of a dehumidifier with residential heat exchangers, I haven't a clue.

The coil temperature MUST be less than the dew point of the air, by the very definition of "dew point".

Indeed, but any amount of cooling will put the temperature below ambient and you'll get some condensation, and the very nature of imperfect conductors (radiators) means that one portion of the coil will be significantly colder, while the rest will be a much smaller temperature difference.

Often, that temp is less than desired for discharge air temp.

Yes, but we're talking about a DEHUMIDIFIER, not an air conditioner. All that heat will be added back.

Call me when you've bought a psychrometric chart and a ductulator.

You're quick to name calling, but it's pretty damn clear you've never even SEEN a dehumidifier. If you're claiming to be an HVAC designer, you'd have to be by far the single most incompetent I've ever seen.

Re:New cooling strategy needed?

[R2.0]

I hate appeals to authority, but this is getting old.

I have a degree in Mechanical Engineering, including a course in HVAC design and a number of them in thermodynamics.
Most of my career has been in building construction for General Contractors and Building Owners, dealing specifically with HVAC system installation and troubleshooting in everything from commercial, industrial, laboratory, and data centers.

You are purposely misreading my statements and attributing meanings that do not fit reality, much less what I was saying.

In my first post, I was talking about free cooling, which is bringing air directly in to a space because it is already cooled. I also responded to your comment "dehumidification only shouldn't necessarily require running a coil with the 20-30 degree temperature gap that the AC coil requires, thus saving some energy.", because it is untrue, and I gave reasons.

You turned it into heat exchangers, and your rebuttal to my comments on dehumidification violate physical laws, and I pointed that out.

Now, it's ALL about dehumidification "This is a brutally stupid statement. Dehumidifiers run the outside air over the cool coils to dehumidify it, then over the hot coils to re-heat it."

No shit - that's what I said before regarding residential units.

"THAT is the heat exchange."

Not really - it's a refrigeration cycle. It uses heat exchangers (refrigerant to air) as components, but the net is heat INPUT into the space.

"You'd be an idiot to run it over the hot coils first, that's the exact opposite of what you need."

And since I was addressing your comments regarding residential outside air heat exchangers, I poined out how stupid that would be. Somehow, when I was talking about outside air heat exchangers, you thought I was talking about dehumidifiers.

"Why you're confusing the simple operation of a dehumidifier with residential heat exchangers, I haven't a clue."

I don't think I'm the one who is confused here. Do you have any qualifications at all to be saying the things you are saying?

I didn't even address these beauties:

"No, what you do is install a second fan."

Regarding the second fan, what do you get? More airflow? If the duct size is the same, your pressure goes way up, and your velocity. So you are burning the same amount of energy - actually more, because large fans are typically more efficient than small, what with motor losses, etc.

"It's a patently ridiculous claim, anyhow. If you couldn't blow-in enough cool air from outside, then you couldn't move enough air to/from the AC units.."

Sure about that? In a server room the is typically very little ductwork - a raised floor acts as a plenum, and short return duct go back to the air handler which is in or right next to the room. If you are bringing air in from the outside, unless the room is near an outside wall there is ductwork required to carry the air, sometimes long runs. The ductwork resists air flow and conducts heat into the airstream, so larger fans and cooler air are required. And you want to run this ductwork for that portion of the year when free cooling is feasible? Especially when those same climactic conditions make conventional cooling system much more efficient? What's the ROI on that?

If you are going to troll, at least know SOMETHING about your topic.

Re:New cooling strategy needed?

[flappinbooger]

Wow, as a HVAC engineer in a former life I got a kick out of reading this "exchange". Pun not necessarily intended.

I think what is ultimately being kicked around here is an "outside air machine" which undoubtedly any data center is likely to employ due to fresh air requirements of most building codes. It brings in raw outdoor air and the building dumps "used" air at a lesser rate, and, the building is then pressurized.

Also, you are right in saying that a typical OA machine must cool the air down to, oh, 50 to 55 F to get decent condensation. Then, it either employs hot gas reheat (if it has hot gas there) or an electric heat coil (for chilled water systems etc) to get the air back to 70F or whatever is required. Omitting reheat in hot climates can save energy by taking load off the AHU's the OA machine serves. Yes there is energy savings to the OA machine when temps are favorable, work is work, but some machines are more efficient, and more intelligent. (my old boss used to ask the OA salesmen "What does your machine do when it's 50 degrees and raining?) .

The thing is, in the real world, you might get something like 4 hours on the third Tuesday of October every other year when the ambient conditions would be perfect. By perfect I mean a suitable temperature and humidity where any sane person would allow the air to be blown directly on their server farm. Way too much risk, all OA must be processed. Side note: OA machines are very expensive per CFM.

Re:New cooling strategy needed?

[afidel]

I doubt you would spend 1/3rd the power to continuously dehumidify air in a once through system then to cool already coniditioned air 20-30 degrees.

Re:New cooling strategy needed?

[evilviper]

You are purposely misreading my statements and attributing meanings that do not fit reality,

Not true in the slightest. If I've misread anything you've written, I suggest you take some time to work on your written skills.

I also responded to your comment "dehumidification only shouldn't necessarily require running a coil with the 20-30 degree temperature gap that the AC coil requires, thus saving some energy."

I have NEVER said anything of the sort.

I was talking about free cooling, which is bringing air directly in to a space because it is already cooled. [...] You turned it into heat exchangers,

Heat exchangers were an example that refute your claims that such methods are "not used in residential," and are "an order of magnitude more complicated."

You can't seem to get your head around the fact that, though not the primary purpose, they do indeed operate along-side the AC, and for "free" happen to bring in a small percentage of cooler outside air.

My further comments had NOTHING AT ALL TO DO WITH HEAT EXCHANGERS. Simply, the operating principles of dehumidifiers.

your rebuttal to my comments on dehumidification violate physical laws

Not remotely true at all. I can back up everything I've said.

Somehow, when I was talking about outside air heat exchangers, you thought I was talking about dehumidifiers.

You quote my statement about dehumidifiers, where I explain why dehumidifiers use far less energy than AC. Then, directly below, you start talking about passive (residential) heat exchangers instead... You think it's my fault you either can't read or quote?

In either case, you still haven't tried to back up your (factually inaccurate) assertion that dehumidifiers consume more power than AC.

If you are bringing air in from the outside, unless the room is near an outside wall there is ductwork required to carry the air, sometimes long runs.

If you aren't "near" either an outside wall or the roof, you're pumping some type of coolant a rather impractically long distance to get to the compressor and back.

Re:New cooling strategy needed?

[swb]

You can buy heat exchangers for homes, in fact they are being required or close to being required for Minnesota due to the very serious problems you mention, all caused by too-tight houses, which we have due to our insulation & energy codes. In fact, we have friends that have one -- it brings in fresh air from outside and processes it, although I think the air volume is more on the slow exchange side and it couldn't be used for cooling.

I'd personally settle for being able to just bring in the outside air when it was below my set point, even if it did bring in humidity; my house will get humidity in it if I open the windows anyway, and any fresh-air system that cycled reasonably frequently would keep it from building up excessively. In the true cooling seasons, it would be eliminated pretty quickly by A/C operation.

I don't know if it would save a ton of money, but right now the alternative is either open all the windows at night and run fans or set the setpoint for A/C too a lower temp to keep the A/C cycling more often. I can't imagine that moving air from the outside would be more energy intensive than running the compressor.

Re:New cooling strategy needed?

[jbengt]

The issue isn't dehumidification of outside air, as, if the outside air is below 55F+/- at 100% humidity, you won't need to dehumidify it, and if the outside air is above 60F+/- you'll still need at least some mechanical refrigeration.
It's when the outside air is really cold, and you're trying to directly cool the data center with it, you could end up with less than 10% RH in your data center, great conditions if you want static electricity. So you end up needing humidification, usually by heating water to produce steam.
Rather than using air directly, most computer room "free cooling" systems use a glycol solution in the chilled water, with a loop to a drycooler on the roof, so you can turn off the chillers in cold enough weather.

Re:New cooling strategy needed?

[jbengt]

"so why not use conditioned external air to cool the equipment and then exhaust it "

Short answer, because the outdoor air is not conditioned.

A simple back-up I've used on some small server rooms is an exhaust fan that draws air from the open offices. The offices will heat up much slower during a loss of air conditioning than a server room, so you'll be pulling conditioned iar through the server room.
But if you are serious about back-up, you need redundant air conditioning systems powered by a backed-up power source.

Re:New cooling strategy needed?

[evilviper]

So you end up needing humidification, usually by heating water to produce steam.

I can't agree with you there. Heating water to produce steam is absolutely the least efficient method of humidifying air, and pretty uncommon, outside of perhaps hospitals and similar reasons of medical-necessity. Other types of humidifiers work quite nicely, while requiring practically no power.

Re:New cooling strategy needed?

[evilviper]

You would be mistaken. The air conditioner and the dehumidifier have to handle an equal volume of air, whether it's in a loop, or a straight-through system.

If you were cooling outside air by "20-30 degrees" and passing it through, it would cost just as much as cooling recirculated air the same amount.

Re:New cooling strategy needed?

[sjames]

Humidity isn't nearly the problem most people make it out to be. Most electronics are perfectly happy as long as the environment is non-condensing and not so dry that static builds up. If a portion of the warmer inside air is recycled into the incoming air from outside, it's not hard to meet the non-condensing requirement.

In cold weather, the problem is not enough humidity. Humidification is a LOT less energy intensive than air conditioning.

I think it would be interesting to actually analyse the cost of additional ductwork and air filters to exhaust hot air from servers to the outside and replace it with conditioned outside air when the outside temperature is sufficiently low.

In the event of A/C failures, even if not ideal, such a setup should provide more safety margin than is available now.

No cooling unit UPS is not too unusual

[CambodiaSam]

After reading the articles linked from previous posts, it looks like the third outage was related to their cooling units not coming back online from the power outage linked to the Semi vs. Transformer battle. I know the units in our data center aren't hooked up to the UPS, but instead are wired directly to the generator in case of outage. I belive this is due to the massive number of additional cells that would be needed to keep up with the wattage requirements. The theory is that if the power goes out, you can live without cooling for the couple minutes while the generator pumps out the first giant plumes of black diesel and revs up to max capacity. We had a similar unplanned test when the local grid had a brownout. Luckily, our units functioned as designed. I wonder if their issues before did more damage to the units than they would have expected...

Re:No cooling unit UPS is not too unusual

[azrider]

After reading the articles linked from previous posts, it looks like the third outage was related to their cooling units not coming back online from the power outage linked to the Semi vs. Transformer battle. I know the units in our data center aren't hooked up to the UPS, but instead are wired directly to the generator in case of outage.

In many large datacenters, the standard hookup is powered through motor generators (external and ups are the supply). The MG's then supply the power to the equipment. The most common failure is that the phasing is not synced on transfer to/from utility power. This is generally very bad when transferring loads on 3 phase circuits, since in extreme circumstances it can cause a 3 phase motor to stall.

In one datacenter I contracted for, the transfer switch failed to sync on return to utility power. This caused a massive shutdown (both IBM and Honeywell mainframes in place). The eventual solution (once the switch was repaired) was to allow automatic transfer to backup but require manual transfer back to utility power (dirty utility power sucks). This was the only way to accomplish 7/9s availability for the datacenter (medical records for 8 hospitals).

Damn dihydrogen monoxide

[wsanders]

They should ban that stuff. (dhmo.org)

Ironic advertisement

[davidwr]

Ah, the dangers of context-sensitive advertising.

Ad on the main page when this article was at the top of the list.

Does "50% off setup" mean you'll only be set up halfway before they run out of A/C?

I've seen this a few times

[petes_PoV]

If your data center's cooling system fails, how long do you have before your servers overheat?

The first occasion was over a weekend (no-one present) in a server room full of VAX's. On the monday when it was discovered, we just opened a window and everything carried on as usual.

The next time was when an ECL model Amdahl was replaced by a CMOS IBM. No-one downgraded the cooling and it froze up - solid. This time the who shebang was down for a day while the heat-exchangers thawed out. It was quite interesting watching the temperature monitors, it took a couple of hours until the temperature rose above the "danger" threshold.

So the answer is either, until you arrive at work (2 days or more), or sometimes a bit more heat is a good thing.

Re:I've seen this a few times

[bstone]

This problem has been around since the dawn of data centers. One bank in Chicago with IBM mainframes in the 60's had battery UPS + generators to back up the mainframes, an identical setup to back up the cooling system, plus one more identical backup system to cover failure in either of the other two.

Re:I've seen this a few times

[petes_PoV]

yes, quite. Any datacentre that relies on utility power and does not have the ability to run everything standalone is at least incompetent - bordering on negligent. Plus, if you buy space in one, without having a backup plan you deserve every bad thing that happens to you.

Over here, there are laws that require certain establishments (i.e. financial ones) to have redundant everything, including locations.

Funny you mention this

[Leebert]

A few weeks ago the A/C dropped out in one of our computer rooms. I like the resulting graph: http://leebert.org/tmp/SCADA_S100_10-3-07.JPG

Re:Funny you mention this

[caluml]

At 17:00 too - just when you're ready to head home.

Re:Funny you mention this

[milgr]

That graph doesn't look bad. It indicates that the high temerature was 92F.
Where I work, the AC in one of the two main labs goes out. I have seen thermometers register 120F. And, the computer equipment keeps running until someone notices and asks people to shut down equipment that is not currently needed.

One of the labs has exterior windows. Once when the AC failed in the middle of the winter, they removed a pane of glass to help cool the lab (this kept the temperature to the low 90's with some equipment turned off). More recently they replaced to plate glass windows with sliding windows so it is easier to open them.

Did I mention that our equipment generates a lot of heat? A couple of years ago 1 fully populated frame required 5 tons of cooling. I think that newer equipment generates even more heat.

Re:Funny you mention this

[Leebert]

That graph doesn't look bad. It indicates that the high temerature was 92F.

Yes, because the A/C came back online. That curve was nowhere near leveling off. There's 200 or so TiB of SATA in that room along with ~1500 ItaniumII processors... :)

Re:Funny you mention this

[R2.0]

Perhaps you should consult with your homophone: www.liebert.com

Really, I thought your link was a spoof site until I read your user name.

Re:Funny you mention this

[Leebert]

Perhaps you should consult with your homophone: www.liebert.com

The units I was talking about are in fact Lieberts (3 50 ton units in that room). Yes, I hear no end to the jokes from coworkers about my nickname.

Re:Funny you mention this

[Calyth]

When our ACs that powers the server room (1 dedicated, and 1 that cools the offices, and supplements the server) both went out, the temperature of the server room rose rather quickly. It hit 30C pretty fast, and we do have a cluster at work.

Raising the set-point, especially for high density computing, is folly, especially for those who needs a cluster, but don't pay someone to babysit the server room 24/7. In that article, if it is indeed the case that a 5kW rack will shutdown within 3 minutes, even having a person present may not be enough to prevent an unplanned shutdown.

Physics

[DFDumont]

For those of you who either didn't take Physics, or slept through it, Watts and BTU's/hr are both measurements of POWER. Add up all the (input) wattages, and use something like http://www.onlineconversion.com/power.htm/ to convert. This site also has a conversion to 'tons of refrigeration' on that same page.
Also note - Don't EVER user the rated wattage of a power supply because that's what it SUPPLIES, not uses. Instead use the current draw multiplied by the voltage (US - 110 for single phase, 208 for dual phase in must commercial blgs, 220 only in homes or where you know thats the case). This is the 'VA' [Volt-Amps] unit. Use this number for 'watts' in the conversion to refrigeration needs.
Just FYI - a watt is defined as 'the power developed in a circuit by a current of one ampere flowing through a potential difference of one volt." see http://www.siliconvalleypower.com/info/?doc=glossary/, i.e. 1W = 1VA. The dirty little secret about power calculations is that there is another factor thrown in, typically about 0.65, called the 'power factor' that UPS and power supply manufacturers use to lower the overall wattage. That's why you always use VA (rather than the reported wattage) because in a pinch you can always measure both voltage and amperage(under load).
Basically do this - take all the amperage draws for all the devices in your rack/room/data center, multiply them by the applied voltage for that device (110 or 208) and add all the products together. Then convert that number to tons of refrigeration. This is your minimum required cooling for a lights out room. If you have people in the room, count 1100 BTU's/hr for each person and add that to the requirements (after conversion to whatever unit you're working with). Some HVAC contractors want specifications in BTU's/hr and other want it in tons. Don't forget lighting either if its not a 'lights out' operation. A 40W florescent bulb means its going to dissipate 40W (as in heat). You can use these numbers directly as they are a measure of the actual heat thrown, not of the power used to light the bulb.
Make sense?

Dennis Dumont

Re:Physics

[timster]

The dirty little secret about power calculations is that there is another factor thrown in, typically about 0.65, called the 'power factor' that UPS and power supply manufacturers use to lower the overall wattage.

It's not "thrown in" by the manufacturers. The dirty little secret is simply that you are talking about AC circuits. 1W = 1VA in AC circuits only if the volts and the amps are in phase -- which they aren't.

Take a sine wave -- in AC, that's what your voltage looks like, always changing. If you're powering something purely resistive like an incandescent bulb, your amps follow the same sine wave and 1W=1VA. But inductive loads like power supplies introduce a lag in the current, so that the amps aren't in phase with the volts. As a result, you cannot naively multiply the RMS volts by the RMS amps to get the average wattage -- you have to take the integral of volts times amps through the curve. And for part of that curve, the voltage and the current flow in different directions, which represents negative power (that is, the inductive circuitry is pushing current back across the wire). As a result of this the overall power will always be less than the volt-amps.

Re:Physics

[EmagGeek]

First, 1 Watt is the movement of energy at the rate of 1 Joule per second, and need not be electrically related at all. A watt is energy per unit time.

Second, power factor is irrelevant to cooling calculations because reactive power does not generate heat, even though it does generate imaginary current in the generating device. This is why power companies bill industrial power based on VAH and not on KWH.

Generators are rated for the magnitude of their output current, not just the real component of it. This is also why power companies try their best to load all three phases equally - because in that case the net instantaneous current out of the generator is zero and the physical forces on the windings and stators is constant and uniform.

Also, most server-class power supplies have power factor correction which adjusts the power factor to 1 by adding shunt capacitance to the input of the supply.

The major point that most people seem to be missing in this dialogue is that a 500W PC power supply does not draw 500W from the wall by simply being plugged in. A PC power supply will deliver only what power is needed by the devices connected to it. For example, my server at home is an X2-4800 with 8 hard disks in it, 4 cooling fans, and a 600W power supply. The total power draw on the server box, two UPS units, the 24 port ethernet switch, the router, the cable modem, and the overhead light, is 262W at CPU idle. Just because it has redundant 500W supplies doesn't mean it's going to draw 1000W just sitting there. I have not measured the power with both CPUs at 100%.

Re:Physics

[hobbesmaster]

Just FYI - a watt is defined as 'the power developed in a circuit by a current of one ampere flowing through a potential difference of one volt." see http://www.siliconvalleypower.com/info/?doc=glossary/, i.e. 1W = 1VA. The dirty little secret about power calculations is that there is another factor thrown in, typically about 0.65, called the 'power factor' that UPS and power supply manufacturers use to lower the overall wattage. That's why you always use VA (rather than the reported wattage) because in a pinch you can always measure both voltage and amperage(under load).

If you don't want to learn anything about AC power, please don't call it a "dirty little secret". Wikipedia for example would be a good place to start.

I'll summarize for you though:
Real Power (P) - has units of W. This is the amount of power dissipated into the "real" (resistive) part of a complex load.
Reactive Power (Q) - has units of VAR (volt-amp reactive). This is the amount of consumed by the reactive (capacitive/inductive) part of a complex load.
Apparent power (|S|, S = P+jQ) - has units of VA (volt-amp). If you plot P and Q on the complex axis, you this is what completes the triangle.
Power factor - ratio between real and apparent power. (P/|S|) For purely resistive loads, this is 1. For purely reactive loads, this is 0.

A UPS is basically just a battery with a bit of supporting circuitry. The battery can only deliver a certain amount of energy per time (some type of power). If this energy is all going to a resistive load, then yes, it would be what you think it is, Watts. This is also the apparent power the UPS can deliver in VA (P=|S|, PF=1). If the load is purely reactive, the apparent power will remain the same in VA, however the real power will be 0. For this reason, UPSs are rated in VA. They can make no guarantee about the amount of real power (W) they can deliver, as it will depend on the load.

Hopefully that makes sense and I haven't confused things further. (power EEs may have something to add/correct, I'm more of an electronics one going off memory)

Re:Physics

[afidel]

If your PSU's have a PF of .65 they absolutely SUCK. Our datacenter has a PF of .89 as measured by our UPS's. This is using IBM xseries and HP Prolients along with a smattering of other systems including a midsized PBX and a Xiotech SAN. My home PC has active PF correction and has a PF of .95. If you had an entire datacenter running at .65 you would be getting a HUGE bill from the power company because they have to use a dummy load at the generating facility to balance out that uneven load.

Re:Physics

[jbengt]

If you can measure the actual draws, you'll have a reasonable number to go by. But if you simply add up the volt*amp ratings, you'll end up with way more cooling than you need, since rated amps is typically a peak draw, somtimes only achieved for an instant at startup, and for cooling you want to use an average power consumption. Usually 50% of rated amp draws is safe, but you really should consult an HVAC engineer to determine your needs.

Anecdote

[steveha]

What happens when the primary, secondary, and tertiary air conditioners all shut down.

http://worsethanfailure.com/Articles/Im-Sure-You-Can-Deal.aspx

steveha

Well once option is to recycle heat

[Timberwolf0122]

In a previous /. article (Ancient fridge)we learned that a sterling engine can run off excess heat, so why not power the cooling system with a sterling engine?

The hot air from the cabinates could be pumped by the Stirling engine to the sterling engine, the work done will lower the air temperature which can then be pumped back to the rack.

Now I realize that a Stirling engine might not be able to extract enough energy to cool a rack in a on-going way, during normal operation it could run in a supplementary capacity with conventional air conditioning but in a power outage it could well buy the extra time needed to either get the chillers running or shut down the servers.

equipment heating is a bête noire for me

[br00tus]

I have worked for over a decade as a sysadmin and have seen firsthand the correlation between temperatures and server failure. I have witnessed two small server rooms melt down to lack of A/C. It is important to me because I know high temperatures will mean more likelihood that I will get a phone call in the middle of the night or on a weekend that a drive, processor or whatnot has failed on a machine.

One thing to consider is if the heat measured outside a box is high, the heat on the surface of the processor is much higher. Even with little fans or heatsinks on them, it doesn't do much, remember, fans and heatsinks don't change temperature they just displace heat - and the heat is attempting to be displaced in an environment of a lot of other boxes trying to displace heat.

In our current data center, run by a respected name, I have measured external temperatures in excess of 100 degrees Fahrenheit on some machines. Machines that run 24/7/365. We have small non-production rooms which have cheap fans that fill up with condensation, and a building staff which is supposed to empty the water when it fills up, but often doesn't.

Sometimes it gets kind of insane - I worked for a Fortune 100 financial company that had tons of money, and had a data center with Sun Enterprise 4000 series servers all over the place - yet the server room was above room temperature, and even more so in certain areas. We had disk and processor/memory board failures all the time, but they never really cared about the room temperature - they spent more time making sure the insides of the fibre optic cables were clean.

I have always brought up my concerns, but management has never really taken them seriously, and then I become overloaded with other work and forget about it as well. The ideal temperature for servers is a few degress above 0 Celsius, or even below 0 depending on the equipment. Meanwhile, if you find a server room where the temperature is below 20 degrees Celsius, you're lucky. It's just one of those things where it is cheaper and easier for them to just waste my time than to fix the problem.

Re:equipment heating is a bête noire for me

[Chirs]

I don't buy the idea of keeping your server room near freezing. Any server that can't handle being in a 20 degree C room is a piece of crap.

Admittedly, this means that you may need rack-specific cooling if some of the equipment puts out more heat than others. We have custom shrouds on some of our racks for exactly this reason.

Short-cycling protection

[Animats]

Most large refrigeration compressors have "short-cycling protection". The compressor motor is overloaded during startup, and needs time to cool. So there's a timer that limits the time between two compressor starts. 4 minutes is a typical delay for a large unit. If you don't have this delay, compressor motors burn out.

Some fancy short-cycling protection timers have backup power, so the the "start to start" time is measured even through power failures. But that's rare. Here's a typical short-cycling timer. For the ones that don't, like that one, a power failure restarts the timer, so you have to wait out the timer after a power glitch.

The timers with backup power, or even the old style ones with a motor and cam-operated switch, allow a quick restart after a power failure if the compressor was already running. Once. If there's a second power failure, the compressor has to wait out the time delay.

So it's important to ensure that a data center's chillers have time delay units that measure true start-to-start time, or you take a cooling outage of several minutes on any short power drop. And, after a power failure and transfer to emergency generators, don't go back to commercial power until enough time has elapsed for the short-cycling protection timers to time out. This last appears to be where Rackspace failed.

Dealing with sequential power failures is tough. That's what took down that big data center in SF a few months ago.

Re:Short-cycling protection

[corsec67]

Easy way to test that:

Have a 2 year old play with the main power switches...

Re:Short-cycling protection

[R2.0]

Not exactly - it has more to do with starting the compresor against full high-side pressure. The critical time preiod is "finish to start", not "start to start".

Short and sweet link.
https://hsb.com/thelocomotive/RecentClaims/FullStory/RC-FS-LOCOCL9.html

Re:Short-cycling protection

[fred fleenblat]

I've seen a lot of consumer-level a/c units that do the delay every time you re-apply power, whether the compressor was on at power off or not. That saves them the cost and complexity of a timer that works while the power is out.

Seems like the whole thing could be avoided by a pressure switch, or a soft start of the motor to see how much (physical) resistance there is before giving it full power.

Re:Short-cycling protection

[TheHawke]

All of the new AC units have compressor protectors built in at the factory. The commercial/industrial units have suction accumulators and liquid line receivers that prevent liquid refrigerant from entering the compressor, causing a "slugging". This event causes damage to the compressor which is costly.
Solenoids are also implemented as well to meter the gases.

A well-designed system should not cycle excessively or run longer than it was designed for. One fine example is one operation was going through a 15,000 BTU window AC a year keeping their wireless equipment cool. The poor window-shaker simply could not stand up to that kind of stress of running 24/7 and it would burn its compressor out. Once we got a larger commercial grade cooling system put in, that is purpose-built for keeping datacenters cool and dry, their bills took a nice nosedive.

Re:Short-cycling protection

[jbengt]

Right and wrong.
Even small compressors often have anit-short cycle timers. This is because the refrigeration cycle needs to be off for a while before you can restart the compressor without overloading it. For example, if I turn off my AC at home, and immediately restart it, I will trip the circuit breaker (I know from experience) So you need to wait that 5 minutes or so to restart, no amount of keeping track of time through the power outage will allow you to skip that 5 minutes of off time.

Re:Liquid cooling on a datacenter level?

[Smidge204]

The solution (no pun intended) is to use something other than water for coolant, such as a fluorocarbon liquid (Not to be confused wth ozone depleting chlorofluorocarbons...)

LFCs are electrically and chemically inert, allowing for direct submersion cooling. If that is undesirable, a normal "water block" heat sink can be used and leaks are only a mess issue, not a functional one.

"best" would be, IMHO, a heat pipe design inside the server module that brings heat to a contact surface on the back, which then thermally connects to a cooling backplate of some kind. This eliminates any liquid inside the modules themselves and mitigates the leak risk completely.
=Smidge=

It could be done.

[Kadin2048]

You could do it, it's just probably more expensive than forced-air cooling.

What you probably would want to do is have a closed system that's actually inside the computer. Fill it with some sort of nonconductive/noncorrosive coolant that won't destroy the machine if it leaks (e.g. 3M Fluorinert), then have a cooling block on the back, away from the electronics, where you plug in the chilled water lines. If you don't daisy-chain, and instead end-run the water intake/exhaust lines from every machine to a central pump, and more importantly than that, you have it driven by suction on the return side rather than positive pressure on the supply side, you could easily attach and detach machines without leaks. (Since in a datacenter a leak is probably more disastrous than a LoC to one server, suction is preferable to positive pressure.)

You'd disconnect the supply from a machine using a quick-release valve; then wait a second for the suction on the return side to pull the water out of the machine's cooling block and start sucking air. Then you'd disconnect the return side. This obviously means that you'd need a way of separating the air out of the return side before it hits the pump, but that's not exactly a unique engineering problem.

It's all doable, but the problems are the expense and the standardization. There's a major chicken-and-egg problem with equipment: you don't want to build a datacenter that can't use commodity equipment, but hardware manufacturers don't want to build gear that can't go into a standard air-cooled rack. So even though datacenters may be the biggest purchasers of racked servers (I'm not sure of that but I suspect they are, at least of some types), and datacenters might be better served by some sort of cooling besides forced-air, everybody gets the lowest common denominator.

Re:It could be done.

[ivan256]

In a closed loop, there is no difference between sucking on the return, and pumping material into the supply. Using the pump to assist draining as you describe would require one pump per server, as the coolant is preferentially going to come from the loops that are being backfilled by the pump itself, and not from the disconnected pipe.

Ideally, you would connect the systems with quick-releases that close off on both sides. That would prevent the need for draining the components inside the system *ever*, and it would also eliminate the need to purge the air from the system.

The reason these types of systems aren't used already is the same as the reason most datacenters aren't supplying our servers with DC power to prevent the loss of 20% of the energy before they even start processing data.... There aren't any standards, and there is a lot of legacy equipment out there. We need intel, and Sun, and a bunch of switch manufacturers to get together and come up with a standard for external DC power supplies and cooling connections. There also needs to be a low cost, small scale box that can produce what those connections need in a small setting for not much more money than the old way. Then over time we'll see more efficiency in this space.

Re:It could be done.

[Kadin2048]

In a closed loop, there is no difference between sucking on the return, and pumping material into the supply. Using the pump to assist draining as you describe would require one pump per server, as the coolant is preferentially going to come from the loops that are being backfilled by the pump itself, and not from the disconnected pipe.

In a totally closed loop, sure. But when you start dealing with leaks it matters. A system driven by suction will draw air in through any leaks; a pressurized system will force coolant out. So if you want to prevent leaks (accepting that you're always going to have imperfect seals somewhere), it would make sense to use suction on the return side of the lines going from the chill units to the servers themselves.

That means the system will tend to leak air in, rather than coolant out.

Re:It could be done.

[d3matt]

The reason these types of systems aren't used already is the same as the reason most datacenters aren't supplying our servers with DC power to prevent the loss of 20% of the energy before they even start processing data.... There aren't any standards, and there is a lot of legacy equipment out there. We need intel, and Sun, and a bunch of switch manufacturers to get together and come up with a standard for external DC power supplies and cooling connections.

You have apparently never heard of the telecom industry. Every piece of equipment they use is DC.

Intel has several platforms (ATCA uses DC exclusively) such as a 2U rackmount chassis that have DC options for the power supply.

As far as externel DC power supplies go, Tyco has a very nice 1U power supply that converts AC to DC. I've also used products by Kepco and APC to get the job done.

Re:It could be done.

[ivan256]

I *have* heard of the telecom industry, and that's exactly why I said "most" datacenters... Unfortunately, that's a telcom standard, and not an industry wide market. There are other features of intel's telecom boxes that make them impractical (far too expensive) for use in the whitebox market.

Re:It could be done.

[ivan256]

Where do you propose the coolant be sucked to, exactly, if the system isn't "truly closed"? Do you want your coolant open to the atmosphere somewhere?

A circulating system may have a pressure variance from one side of the pump to the other based on fluid viscosity, pipe diameter, and flowrate, but in a well designed system that would be negligible.

If you did design an open circuit system, and managed to develop sufficient negative pressure, you'd still end up with air in your circuit in the event of a leak. That would be *bad*, since it wouldn't cool your equipment correctly. Especially if it forms a cavity in the system somewhere. The best way to deal with leaks is not to have them. With well designed and implemented joins in your plumbing, this shouldn't be all that difficult. We've solved this problem over a century ago.

Re:It could be done.

[turbidostato]

"Do you want your coolant open to the atmosphere somewhere?"

Not necesarily. But you certainly will have an expansion chamber somewhere that will take out (minor) air "leakeages" as well as any other volume variations. Pression excess will go out through a one-way valve on top of the expansion chamber as needed.

"you'd still end up with air in your circuit"

That's another reason for an expansion chamber (as well as purge valves through the route). Once the air in the expansion chamber it will simply "bluff" away.

"The best way to deal with leaks is not to have them"

The best way to deal with leaks is to know you *will* have them no matter what, avoid the obvious (and not so obvious as possible) and plan for the worst.

"We've solved this problem over a century ago."

Still I don't know of any circuit no matter how "closed" it's designed that it doesn't take into account leakeages, purging and volume variations.

Why run data centres in hot states?

[EmbeddedJanitor]

It seems crazy that the data centres seem to run in hot states. Surely Alaska would be better? C'mon Alaska, get the tax-breaks right.

Re:Why run data centres in hot states?

[arth1]

(Disregarding your blatant karma whoring by replying to the top post while changing the subject)

There's several good reasons why the servers are located where they are, and not, say, in Alaska.
The main one is light speed through fiber, and a cable from Houston to Fairbanks would induce a best case of around 28 ms latency, each way. Multiply by several billion packets.

This is why hosting near the customer is considered a Good Thing, and why companies like Akamai have made it their business of transparently re-routing clients to the closest server.

Back to cooling. A few years ago, I worked for a telephone company, and the local data centre there had a 15 degree C ambient baseline temperature. We had to wear sweaters if working for any length of time in the server hall, but had a secure normal temperature room outside the server hall, with console switches and a couple of ttys for configuration.
The main reason why the temperature was kept so low was to be on the safe side -- even if a fan should burn out in one of the cabinets, opening the cabinet doors would provide adequate (albeit not good) cooling until it could be repaired, without (and this is the important part) taking anything down.
A secondary reason was that the backup power generators were, for security reasons, inside the server hall themselves, and during a power outage these would add substantial heat to the equation.

Re:Why run data centres in hot states?

[magarity]

It seems crazy that the data centres seem to run in hot states
 
Several people have asked this and you're all thinking inside the box; a better idea is why not locate data centers near large cavern complexes or abandoned mines where the air temp inside is in the 60's no matter what the temp outside. Run a duct into the cave and you have natural AC. Just need a fan and if the power to that goes out, have one of those exercise bikes with the big fan as a wheel for the backup.

Re:Why run data centres in hot states?

[arth1]

While thinking outside the box is all well and fine, it's even better when combined with Common Knowledge. Like knowing that caves and mines (a) tend to be rather warm when deep enough, and (b) have a fixed amount of air.

As for the power efficiency of pumping air from several hundred meters away compared to pumping it through the grille of an AC unit, well, there's a reason why skyscrapers these days have multiple central air facilities instead of just one: Economics.

I'd like to see you pump air for any long distance with your exercise bike :-)

Re:Why run data centres in hot states?

[phantomcircuit]

The heat generating oil burning back up generators were in the air conditioned room?

Seems a bit foolish to me...

Re:Why run data centres in hot states?

[Spazmania]

the local data centre there had a 15 degree C ambient baseline

Well that's just incompetent. For one thing, commercial electronics experience increased failure as you move away from an ambient 70 degrees F regardless of which direction you move. Running them at 59 degrees F (15 C) is just as likely to induce intermittent failures as running it at 80 degrees F.

For another, you're supposed to design your cooling system to accommodate all of the planned heat load in the environment. If your generators will be adding heat then the A/C needs to have sufficient capacity to take that heat back out.

And anyway, your generators shouldn't be adding heat. They should be walled off from the data center with exterior air exchange. Otherwise an error in the exhaust ducting risks killing your operators with CO poisoning.

Re:Why run data centres in hot states?

[h4ck7h3p14n37]

The main reason why the temperature was kept so low was to be on the safe side

There's a Google study that suggests, for disk drives, a higher temperature does not correlate with higher failure rates. However lower temperatures do correlate with an increase in failure rates.

Figure 4 shows the distribution of drives with average temperature in increments of one degree and the corresponding annualized failure rates. The figure shows that failures do not increase when the average temperature increases. In fact, there is a clear trend showing that lower temperatures are associated with higher failure rates. Only at very high temperatures is there a slight reversal of this trend.

"Failure Trends in a Large Disk Drive Population"

It appears that for modern disk drives, cooler is definitely not better.

Re:Why run data centres in hot states?

[arth1]

Well that's just incompetent. For one thing, commercial electronics experience increased failure as you move away from an ambient 70 degrees F regardless of which direction you move. Running them at 59 degrees F (15 C) is just as likely to induce intermittent failures as running it at 80 degrees F.

Note that 59 F ambient temperature doesn't mean that the components run at an operating temperature of 59F. About the only components subjected to the 59F temperature would be cables and intake fans.

Re:Why run data centres in hot states?

[arth1]

And anyway, your generators shouldn't be adding heat. They should be walled off from the data center with exterior air exchange. Otherwise an error in the exhaust ducting risks killing your operators with CO poisoning.

In inner city installations, that isn't always an option. Anyhow, the halon extinguishing systems pose a much greater threat to operators (who shouldn't be in the server halls anyhow, unless it's to make physical changes, which would normally be suspended during a power outage).

Re:Why run data centres in hot states?

[dbIII]

Internal walls are cheap. Roof space is cheap. Also a lot of stuff often has to be done during power outages, there's often unexpected things that require intervention or it's just a really good time to clean up the cabling. Sometimes circumstances dictate really stupid designs but that doesn't remove the stupidity. The CO problem is also a major reason to put such stuff outside somewhere since apparently it is not something you can be aware of as it happens. Two guys in a place where I started work the next week just dozed off and never woke up. Even if you lock the area off and do air testing before sending people in (since we now have an industrial confined space instead of a simple server room) there is the potential problem of great big lumps of soot getting into the drives.

When you have people playing games of instant DIY instead of calling in the engineers or tradespeople you get stupidity like putting great big smoke belching heat sources in rooms where you want clean cold air. Sometimes putting in a wall and removing some windows is the difference between abject stupidity and a well run system.

Re:Why run data centres in hot states?

[dwater]

There's a Google study that suggests, for disk drives, a higher temperature does not correlate with higher failure rates. Google don't seem to make the same conclusion as you. From the same report, page 5 "Conclusion" :

"One of our key findings has been the lack of a consistent pattern of higher failure rates for higher temperature drives...Although our data do not allow us to conclude that there is no such correlation,"

Instead, a direct quote (from the same sentence) uses it in a different context :

"it provides strong evidence to suggest that other effects may be more prominent in affecting disk drive reliability".
Google don't seem to make the same conclusion as you.

From the report, page 5 "Conclusion" :

"One of our key findings has been the lack of a consistent pattern of higher failure rates for higher temperature drives...Although our data do not allow us to conclude that there is no such correlation,"

Instead, a direct quote (from the same sentence) uses 'suggest' in a different context :

"it provides strong evidence to suggest that other effects may be more prominent in affecting disk drive reliability".

Re:Why run data centres in hot states?

[RockDoctor]

the local data centre there had a 15 degree C ambient baseline

Well that's just incompetent. For one thing, commercial electronics experience increased failure as you move away from an ambient 70 degrees F regardless of which direction you move. Running them at 59 degrees F (15 C) is just as likely to induce intermittent failures as running it at 80 degrees F.

I was considering asking why the GP poster was bothering with a sweater when working (as opposed to sleeping) in his server room at 15centigrade, but decided that he must just be one of those people who can't stand normal temperatures. But electrical engineers know that a lot of their equipment is going to be used in "ambient" conditions which are not the "ambient" of their climate-controlled office. In my work, for example, 20C would be an abnormally hot temperature for our sensor equipment ; -20C would be by no means unknown; -50C quite credible. On the other hand, some of our analytic equipment has to run for months between service visits at +50C in 90%+ condensing humidity and with forced ventilation carrying salt dust and oil spray. You design your equipment for the conditions that it's going to face, not the conditions in your office today.
Additionally, you appear to be conflating the air temperature in the data centre (15C) with the temperature of the components. Since having a heat flux requires having a thermal gradient, then the components will be warmer than your heat sink.
In this town, we can tell the nationality of the boss of any office instantly on walking in - European bosses keep the HVAC (heating ventilation air-conditioning, or climate control) set to about 20C ; American bosses have it re-set to 25C (until over-ruled for wasting money).

For another, you're supposed to design your cooling system to accommodate all of the planned heat load in the environment. If your generators will be adding heat then the A/C needs to have sufficient capacity to take that heat back out.

There's an Indian HVAC company (in Abu Dhabi), and a instrumentation engineer (last heard of in Houston, America) who need to be taught this lesson. Again. If you meet them, please apply the clue-bat before agreeing to take the equipment they design out to the Empty Quarter to rig it up.

[your generators] should be walled off from the data center with exterior air exchange. Otherwise an error in the exhaust ducting risks killing your operators with CO poisoning.

Your carbon dioxide flood for fire suppression would be as effectively lethal. Operators would need to be kept out of the controlled zone while enclosed generators are running; the fire suppression system should be overridden while operators are in the controlled zone, or you need to be rigged up with cascade air supplies and work-pack SCBA while working in the control zone. This isn't rocket science - there are plenty of corpses that point the way to proper management of work in potentially lethal atmospheres. (Of course, there are plenty of work places that like to cut corners and put their workers at risk. Don't work there and do report them to the relevant authorities.)

Re:Why run data centres in hot states?

[flurdy]

It seems crazy that the data centres seem to run in hot states. Surely Alaska would be better? C'mon Alaska, get the tax-breaks right. I was just thinking the same.

This is why hosting near the customer is considered a Good Thing, and why companies like Akamai have made it their business of transparently re-routing clients to the closest server. I can see why some servers and therefore data centres needs to be close to the companies that owns them. Certainly always been the case with the companies Ive worked for. Call out time is short and network lag between other apps/dbs are low and response time quicker.

However most servers do not really need to be next door. My server is in a different country, and there is no notable difference. Most web sites ( unless perhaps video heavy ) would fine anywhere in the world, especially the load balanced redundancy servers. Alaska, Scandinavia etc just open the door if gets too varm. (Ok, not pedantically, due to dust, extreme temperature differences etc.)

Think how much CO2 we would be saving by filling up Iceland with data centres... ;)

Re:Why run data centres in hot states?

[budgenator]

This is not possible when there's a -10F blizzard outside. Shit man what would they do in the winter when it gets cold, in a lot of places -40 -- -50 are typical winter temps.

Re:Why run data centres in hot states?

[sglines]

About 10 years ago I worked in a data center that lost cooling. The main water pipe to the cooling towers burst flooding the subfloor. All the old VAX systems overheated immediately and shutdown. Next to fail were the numerous PC based Unix systems (SCO mostly) then the HP, DG and IBM systems. None of the Sun systems ever overheated. The temperature in the server room got up to 120F before someone got the good idea to open some windows. The data center occupied a full floor of a modern glass office building. The windows had to be unscrewed and the plate glass carefully removed with suction cups. I was amazed that someone had actually thought of this. Fortunately it was February and the outside temperature was in the low 20's F. It took almost 24 hours to get everything cool enough to get back online.

Re:Why run data centres in hot states?

[sjames]

In some places, putting a generator IN a server room is a serious code violation because of the CO problem.

CO2 from a fire suppression system is FAR different than CO from a generator or another inert gas fire supression system. The breathing reflex in humans is triggered by the CO2 level in the blood rather than the oxygen level. If a load of CO2 is dumped into the room, the occupants will feel respiratory distress and know to get out. If the fire supression system dumps a load of inert gas, there will be NO respiratory distress because the CO2 will be exhaled normally. Fortunatly, there's generally incredibly loud alarms telling you to get out and small amounts of the gas from a leak are harmless.

CO is much worse. It not only replaces the oxygen in the room but also poisons the hemoglobin so that even getting the person to fresh air may not be enough. In the worst cases, only hyperbaric oxygen can help if anything can.

AC's use Power???

[spiedrazer]

I fail to see where this could be news to anyone who works with data centers. If you want your datacenter to operate during a power outage, you need a Generator with enough capacity for your servers/network and your cooling. If a fancy hosting site with SLA's making up-time guarantees doesn't understand this, I think thier customers should start looking elsewhere.

Re:AC's use Power???

[bizitch]

Thats the first that came to mind for me as well -

What? No freaking generator? Umm....wtf?

If not a matter of IF you lose power - just WHEN you lose power

It will happen - I guarantee it

Re:AC's use Power???

[kmankmankman2001]

I'm still not clear (and I DID read the article) how their basic power feed is configured. The followup article references that they were switching over to the backup power feed. Why aren't BOTH feeds configured so that when one of them goes poof there's no interruption? All of our data centers have multiple feeds coming in from different grids into different entry points in the buildings and running up different risers until they meet at the distribution point. No single point of failure. And when ComEd craps out (it's a when, not an if, but they are MUCH better than they were not too long ago) it's the generators to the rescue. However I don't believe the HVAC is on the UPS, either, for the same reasons others have posted - so there probably would be an interruption there if we switch to diesel. Something to check into as we are actually in the process of planning (yet another) data center.

Backup Generator?

[dbdavids]

The real solution here is not really a multihour ups for cooling and power, it would be a emergency generator. Have the generator auto kick on after 5 minutes and have 10 minutes of UPS time on the equipment. Generally I have found there is no need to ups the cooling unit.

Highlights Serious Flaw - Neglecting Outside

[Ron Bennett]

While many here are discussing UPSes, chillers, set-points, etc the most serious flaw is being glossed over ... the lack of redundency outside the data center, such as multiple, diverse power lines coming in...

From the articles, it appears that Rackspace datacenter doesn't have multiple power lines coming in and/or many come in via one feed point.

How else is it that a car crash quite some distance from the datacenter can cause such disruption. Does anyone even plan for such events - I get the feeling most planners don't, since I've seen first-hand many power failures occur in places where one would expect more redundency from dumb things like a vehicle hitting a utility pole, etc.

Ron

Re:Highlights Serious Flaw - Neglecting Outside

[PoderOmega]

From http://www.rackspace.com/information/announcements/datacenter.php/

In the second incident at approximately 6:30 PM CST Monday, a vehicle struck and brought down the transformer feeding power to the DFW data center. It immediately disrupted power to the entire data center and our emergency generators kicked in and operated as intended. When we transferred power to our secondary utility power system, the data center's chilling units were cycled back up. At this time, however, the utility provider shut down power in order to allow emergency rescue teams safe access to the accident victim.

It looks like you are correct. They refer to a "secondary utility power system" but it appears to be going through the same transformer/transformer park. Its kind of like our ISP trying to sell us a second "backup" T1 at the office, when they are both on the same provider. There could be some specific situations where it would work as a backup, but you still have the same weak link in the chain (the provider or backbone goes down). I am not sure why Rackspace had there power set up this way. Maybe it was the cheapest or only option. Maybe it was the cheapest way they could claim they have a backup utility, and still be honest. I wouldn't think to ask "Oh, you have a backup utility, do they both go through the same transformer/transformer park?"

Re:Highlights Serious Flaw - Neglecting Outside

[PPH]

You have to pay for redundant feeds from the local utility company. And they aren't cheap. If you don't select a location on the boundary of two independent distribution circuits, the two feeds are worthless.

I live near a hospital which is located on the boundary between two distribution circuits, each fed from a different substation. That redundancy cost the hospital tens or hundreds of thousands of dollars. But the two substations are fed from the same transmission loop, which runs through the woods (lots of trees and on inaccessible rights-of-ways), so the most probable fault will take both stations, circuits, and sources to the hospital off line.

The moral of the story: Don't depend on an outside organization (the local utility) for service when its your neck on the line and not theirs.

Re:Highlights Serious Flaw - Neglecting Outside

[eth1]

I live in the Dallas area, and heard a little about this incident on the local news. It was vague information, but from what I heard, several thousand customers got knocked out with this outage. That would seem to indicate that this wasn't the small power pole on the corner that got hit, but a much larger line, maybe feeding a substation that took out the whole area. To get a redundant line in that situation could be difficult, as you'd have to run a dedicated line in from a separate substation somewhere. That could be extremely hard to do and very expensive in a built-up area like Dallas.

Re:Highlights Serious Flaw - Neglecting Outside

[akpoff]

According to our account manager at RackSpace they do have multiple power lines coming in. When the transformer was damaged everything, including the chillers, seamlessly switched over to the generators. Once all that was working RS moved two of their chillers over to the redundant power line. The local power company then cut power to that line.

According to RS, even though that line was separate from the transformer the power company had to shutdown power to it to safely remove the driver of the truck from truck/power-supply mess. The chillers went down, and has other have noted, the chillers can't be restarted multiple times sequentially. They had to be allowed to reset. In that time heat was escalating fast. Their only choice was to begin taking down servers.

Obviously the decision to move the two chillers over to the secondary power line was premature, but in all likelihood the techs in the building didn't know the power company was going to cut the other power line. Sadly they got caught in an almost "perfect storm" of events.

Re:Highlights Serious Flaw - Neglecting Outside

[chris.dag]

I think you may be underestimating how hard this actually is ...

How many different power grids feed your company?

Redundant networking is well understood, so is the engineering required to bring in that networking via multiple trenches and facility penetration points so that you are protected against someone cutting a line or opening a trench accidentally with heavy equipment.

This is much harder for electricity.

At some point, probably pretty close to your facility you are going to be drawing from the same grid / same source. Whats the point of multiple trenches for power when you are drawing off the same grid or transformer a short distance away? You still have a potential single point of failure. This is why the facilities invest in local power generation and backup rather than diverse feeds which may be impossible or impractical to obtain.

The only reason I know this is that I've got a colocation cage rented at a Boston area datacenter that through an odd stroke of history actually IS connected simultaneously to 2 different utility power grids. The people who work there love to point out how rare it is to find a building that is attached to completely different upstream power sources.

Anyway, just my $.02

Re:Highlights Serious Flaw - Neglecting Outside

[ysachlandil]

There is no lack of redundancy in powerlines. The servers all stayed up when the transformer went out. The problem is that AC units are normally not connected to the UPS, since that would increase power-draw to twice normal and result in the UPS failing in half the time. They are connected to the generators however so when outside power fails, the AC units are offlined for only ten minutes. But that is enough to melt the servers nowadays.

That and the fact that the powercircuits are not tested for failures regularly (unlike the generators) leads to problems like this. But testing the powercircuits is a very difficult thing to do. Inducing failures in loaded circuits that dissipate 200 kilowatts or more is no fun, and the transients are hell on everything connected.

--Blerik

Maxwells data center

[techpawn]

We've summoned a small demon to let in cool air particles and shunt out hot ones. Sure the weekly sacrifice gets to be a pain after a while, but there's always a pool of willing interns right?

computers convert 100% electricity to heat

[mwilliamson]

Every single watt consumed by a computer is turned into heat, and generally released out the back of the case. Computers behave the same as the coil of nichrome wire as is used in a laundromat clothes dryer. (I guess a few milliwatts gets out of your cold room via ethernet cables and photons on fiber)

5kw? ow.

[MattW]

5 kilowatts is a heck of a lot to have on a single rack - assuming you're actually utilizing that. I recently interviewed a half dozen data centers to plan a 20-odd server deployment, and we ended up using 2 cabinets in order to ensure our heat dissipation was sufficient. Since data centers are usually supplying 20 amp, 110 or 120v power, you get 2200-2400 watts available per drop; although it's considered a bad idea to draw more than 15 amps per circuit. We have redundant power supplies in everything, so we keep ourselves at 37.5% of capacity on the drops, and each device is fed from a 20amp drop coming from a distinct data center pdu. That way even if one if the data center pdus implodes, we're still up and at 75%- capacity.

Almost no data center we spoke to would commit to cooling more than 4800 watts of power at an absolute maximum per rack, and those were facilities with hot/cool row setups to maximize airflow. But that meant they didn't want to drop more than 2x20amp power drops, plus 2x20 for backup, if you agreed to maintain 50% utilization across all 4 drops. But since you'd really want to maintain 75%- even in the case of failure, you'd only be using 3600watts. (In the facility we ended up in, we have a total of 6 20 amp drops, and we only actually utilize ~4700 watts.

Ultimately, though, the important thing is that cooling systems should be on generator/battery backup power. Otherwise, as this notes, your battery backup won't be useful.

Re:5kw? ow.

[NetJunkie]

5Kw isn't that much anymore. We are deploying dense environments that are 25Kw per rack and have plans to go above 30Kw. The old hot/cold row methodology has to go away to more efficient designs.

Datacenter cooling should be on generator/ups

[ndege]

The one, seemingly obvious, question I have is, why aren't the cooling needs on generator/ups backup?

I have toured data-centers where even the cooling was on battery backup. The idea is that the battery banks hold everything as it is until the generators come fully online (usually within 30 seconds). The batteries/UPS transformers were able to hold the entire system for approx 30 minutes on battery alone irrespective of generator status. This also reduced the issues from quick brown-outs...no need to fire up the generator for a quick 2 second outage.

Why aren't data-centers like this built to be completely self-sufficient or autonomous?

Re:Datacenter cooling should be on generator/ups

[Critical Facilities]

If you've actually seen cooling systems on UPS, please let me know who those Data Center operators are, because they need to be reported to the authorities...or at least a mental institution. There is absolutely NO reason to put massive, inductive, motor loads on your poor UPS system. By doing so, you are putting your actual critical load in major jeopardy (by exposing them to the potential of electrical faults on the cooling systems) and by reducing your available redundant UPS power.

You can set time delays into your Energy Management Systems and transfer switches to prevent generators from being fired up for 2-3 second brown outs. There is just NO reason to put cooling/HVAC on UPS, it would be insane to do so.

Re:Datacenter cooling should be on generator/ups

[TooMuchToDo]

Almost all datacenters have their cooling needs sized to work on the generator. You can't run the cooling system off the UPS, which is only sized to carry you from utility power to generator power. Only essential loads (i.e. routers, switches, customer equipment) should be run off of UPS power, as the cooling system should come back up when the generator farm fires up. You should be on UPS for NO LONGER then 3 minutes, as generators only take about 60-120 seconds to start up, and your UPS run time is most likely only 10-15 minutes (sometimes less).

Re:Datacenter cooling should be on generator/ups

[Skapare]

A large data center should not have one big massive UPS anyway. It should all be divided out into various load sections, each with its own UPS+battery system. Once you do that, then you can have cooling on its own UPS without any risk of the cooling system impacting the UPS feeding the computers ... if you really want cooling on UPS (it can be done, but generally is not the best way). Surely you would have the cooling on it's own three phase circuits.

Perhaps a better approach is a smart cooling system that rotates the starting of compressors on various units so you always have some number of units running and some number not running, at the ratio needed for the current thermal demands. Then where there is an outage that has to go to generators, only a limit number of units will have been recently started just before the outage and need to be thermally protected. The controller skips those and starts the idle units (unless you are already maxxed out in which case you'd have no idle units). But you will need to have the cooling on the generators.

If you are going to have a backup distribution circuit from the utility, it should be physically separate from the primary circuit so that it is not necessary to shut down both to deal with things like a traffic accident.

Re:Datacenter cooling should be on generator/ups

[Critical Facilities]

I agree with almost all of your post with the only exception being the cooling systems on UPS. There is absolutely no reason to put cooling systems on UPS power. Large, inductive loads are a UPS's enemy. A big inrush current of a chiller starting up would beat the crap out of your battery string(s).

Having said that, you are exactly right on having both your UPS system(s) and your cooling system(s) diversified. I tend to get into this argument with people regarding what constitutes a "data center" and one of the most significant parts of determining what actually constitutes a "data center" is redundancy. This means not just redundant utility power feeds, but redundant UPS systems/modules, redundant generators, redundant chillers/CRACs, redundant PDU's, etc etc.

For our cooling systems, we have 4 Chillers (we only need 2) and 20 CRACs (we only need 10. Any problems with any system can be mitigated by rolling to the redundant system.

My Specific Answer

[rtobyr]

There are a lot of variables that go into this, so here goes:

1. Due to facilities issues (we are a State agency in a County controlled facility), our "data center" is a 6'x11' closet, so 66' sq.
2. The sum of our equipment wattage in the "data center" is min=6510W max=16833W. We estimate the average running wattage to be around 11000W.
3. Assuming 1W/h=3.414 BTU/h, our "data center" generates 22225 BTU/h, 37554 BTU/h, 57468 BTU/h, min, avg, max respectively.
4. Due to the same facilities issues in #1, we keep the "data center" at 71 degrees F, and cannot keep it any cooler.
5. Due to the same facilities issues in #1, we have lots of cooling outages, and therefore much experience that qualifies me to accurately answer the question.

If cooling goes out in our "data center," the servers overheat in 15-20 minutes (when the closet reaches about 115-120 degrees F). To increase this time to about 45 minutes, we have installed a portable cooler that kicks in when the main HVAC system fails.

A tale of redundancy

[kilodelta]

Back when I worked for the Secretary of State's office it was decided that we should move the Central Voter Registration System to a new location, along with all the other mail, web, and database servers.

I was responsible for infrastructure planning. We ended up using an APC Symmetra backed up by a 125kW natural gas fired generator. Transfer to generator took approximately ten seconds by the Symmetra could keep the whole thing running for 45 minutes, allowing us graceful shutdown, etc. if the generator didn't spool up. We even extended power to the MDF so that Cox could plug their UPS into the generator line and be powered up while everything else was down.

The cooling was accomplished by redundant systems. There were duplicate two-ton air conditioning units in the room. If one failed the other could pick up the slack but in normal mode they both ran.

This represented a big improvement over what we'd had before. The servers had been housed in an minimally air conditioned closet in the sub-basement of the State House.

The fly in the ointment so to speak was that we depended on another state agency for DNS service. One day there was a massive power outage that affected a good chunk of Providence,RI. The Sec State's office systems were up and running except nobody could get to them, and they couldn't get out.

And it's still that way today.

Re:Liquid cooling on a datacenter level?

[afidel]

They sort of make what you are describing, many datacenters have glycol cooling and companies like APC make spot cooling solutions which allow you to bring the glycol loop directly to cabinets with unusually high power densities such as a rack full of fully populated blade enclosures. Now they are generally used to cool incoming air for normal systems because the laws of mass production means it is MUCH cheaper to buy a new cooling loop for the rack then it is to buy a rack full of specialized servers.

Power + Heat + Data Centers: a tough problem

[markjl]

Disclaimer: I work with SGI, so I can shed some light on their customer's perspective (NASA, gov't, research labs, etc.) and solution to this problem.

The increasing density of servers is exacerbating the problem of power and cooling in every data center. This week is the SuperComputing trade show where the the new top 500 supercomputers edition was released with "Big Turnover Among the Top 10 Systems," where you can see the first examples to address these issues.

SGI's new ICE blade system was launched a few months ago, it was designed to address the power consumption, real estate density, and cooling issues everyone will probably experience on their next server cycle. ICE has shipped and one installation is now #3 on the Top 500. It's a welcome sign that SGI is back from bankruptcy. I'm sorry if this seems like an advert, so I'm not going to link to SGI -- you can go find out more easily if you want.

Re:Liquid cooling on a datacenter level?

[couchslug]

Go aircraft hardware, and be prepared to pay the price.

Stainless hard lines with short braided Teflon-lined hoses having quick disconnects (HAVE QD PLUGS/CAPS IN CASE THE QD CHECK VALVE LEAKS) could connect the cooling manifold to the servers. I'd use ozone-friendly refrigerant instead of liquid so leaks don't damage hardware. System pressure would be easy to maintain. Have a recovery suction line to each server so it could be emptied after being isolated.

Chilled compressed air would be another, simpler option. The condensation wouldn't bother aircraft hardware, and a large air tank would compensate for pressure variations.

A reason that 18-wheelers use air brakes instead of hydraulic ("liquid") brakes is that it facilitate quick disconnection and tolerates some leakage.

Redundancy... with redundancy to the redundancy

[goobenet]

When i worked at (and help build) a datacenter back in 2000, they planned for just about everything. Then they played the whatif game for a few more scenarios, and came up with the fact that they needed to double the main capacity, and add a second layer. By that i mean that if the main transfer switch on the genset failed, there was a second one that'd kick in. If one or 2 (or all 3) of the A/C units failed for some reason, there was #4 and #5 which could carry the load with acceptable tolerances.

We also had the genset in parallel to the UPS. So when the UPS got down to around 60%, the genset would kick in, and power them back up to 85% and shut down again. There was also 2 more gensets on site for just A/C and other essential systems. (We also learned the hard way how A/C compressors really dislike genset power, and had to get a very expensive line smoother from the power company to make the bigger ones play nice.)

Someone did the math on this system once. The datacenter could sustain itself without outside power for 51 days(!!) running at 70% capacity.

Not to mention, we ran the datacenter once a month for 6 hours on backup systems just to test. Then again, my boss at the time was the type of guy who would pull a card out of a router to see if said router (at that point in time, a Cisco 7513) was hot swap... Live router, had our SONET stuff running through it... Luckily it is a hot-swap router. :)

raising the temperature set point - will not work

[John Sokol]

As computer get warmer there power consumption goes up.

http://www.silentcomputing.com/tech/market2.gif

Using Intel mother boards we found for each 10 Deg C temperature increase there was a 2% power increase.

I spent three years trying to get a company off the ground after I solved how to fix the heat problem.
I realized, they don't want to fix it, unless it's going to make them money.

Re:This isn't rocket science

[PerspexAvenger]

Because:
a) It's turned off, as they want performance.
b) It doesn't apply to the other big heat-maker, the drives.

It's also a lot easier to cool and recirculate air, than to properly clean, (de)humidify and cool external air on the fly.

Re:Better yet a new power strategy

[evilviper]

The powersupply probably generates 25-50% of the heat from a device/server.

50% is outlandish. The number is probably 15-40%. But most importantly, you can't eliminate that loss by switching to DC. DC power supplies aren't any more efficient than AC PSUs. In fact, the two are largely identical.

And you simply can't run computers on a single voltage, or even a small number of voltages. Your CPU and other chipsets need anywhere from 0-3 volts these days. Motors, in hard drives, case fans, etc. are more efficient with the use of higher voltages. Different buses and ports require a range of different voltages. etc.

And even if you could standardize on a single voltage, you'd still need a power supply, because computers won't handle the voltage spikes and surges of a shared line, they need ultra-clean DC, so expect to pay ungodly amounts of money for ultra-massive banks of capacitors and voltage regulators, all with really no gain in efficiency.

How about timeshifted cooling

[Isaac-1]

Last night while spinning the channels on tv I saw a discovery channel/PBS show regarding a library in AZ using ice as a way to timeshift their power demands away from peak power load times. This system used a grid of refrigeration coils imersed in a tank of water, at night when power demands were lower, the A/C compressors would circulate refrigerant through the water tanks, freezing the water into ice, in the day they would dump the heat into the ice letting it melt. It seems a similar strategy would work for data centers for short term banking of lower powered cooling needs.

Ike

Re:How about timeshifted cooling

[fred fleenblat]

here's a some links:
http://www.popsci.com/popsci/flat/bown/2007/green/item_65.html
http://www.ice-energy.com/Default.aspx?tabid=163

Shut up, both of you

[Anonymous Coward]

And stop running the SETI client, install the power scaling software of choice for your architecture, and stop wasting so damn much electricity to keep your precious snowflake busy 24/7.

Re:Shut up, both of you

[ByOhTek]

WTF, I was discussing the heat generated by computers, and giving an example of how it can be used for heating a house, rather than using extra energy for a furnace. If you had a clue you'd notice my last mention of memory wouldn't even *fit* in a computer like I described - it was simply a point on energy use.

No SETI, no Snowflake (whateverthehellthatis), and unfortunately that computer is too old to have much in the way of decent power scaling.

Get over yourself.

I was hit by this outage...

[CompMD]

I have a server in the DFW1 Datacenter that was knocked out during the first two outages but survived the third. I was down for about two and a half hours during the first outage and fifteen minutes during the second. My Rackers answered every question I had about it with honesty and humility. They admitted that this was their problem and kept me informed as to what was going on.

I'm wondering how many people posting on this thread are actual Rackspace customers. I can't say that I have ever once experienced anything less than top quality service from them. They are friendly, knowledgeable, hard working, and I respect them very much, and that's coming from a sysadmin/one man IT department.

Here's a company that is taking responsibility for its actions right off the bat, apologizing for their shortcomings, and honestly trying to make things right for those who have been affected. Think about this in contrast to the corporate scandals and craptacular customer service that has been plaguing the US lately.

Re:This isn't rocket science

[ZwJGR]

That is to stop the CPU chip from permanently destroying itself by overheating (which it would do within seconds if the heat sink was removed, it could last a few minutes without a fan).
It will not affect the heat production of the rest of the system (PSU, motherboard chipsets, drives, etc.)
Furthermore, the performance drop when CPUs do this is so horrific you would do better to pull the plug on the machine, that way there a much more significant heat production drop.
Turning off a section of your machines is better than having them all/more of them effectively hung, and still producing heat...

Combine cooling and UPS

[IvyKing]

Why not use the power of the expanding nitrogen gas to provide power as well - and yet another use is to put out fires as well.

Re:This isn't rocket science

[kmankmankman2001]

There's more than servers to consider. DASD doesn't like to get hot - at all. The motors used in the HDA's are not variable speed so they can't be slowed down, they are either on or off. Running DASD hot for any prolonged period of time is a good way to become familiar with the term " . . . may produce unpredictable results" (and none of them are good).

Re:Energy efficiency

[kmankmankman2001]

Interesting. Of course now we have the benefit of having cut out the middleman - by positioning the data center near the highway the truck doesn't have to bother hitting the transformer it can just ram directly into the data center. :)

Personal horror story

[Tablizer]

I had an ISP that hosted my MySql-based app. The AC broke in their server room and the hard-drives got fried. They restored from the most recent backup, but it appeared they loaded from MySql dumps.

For some reason, auto-incrimenting columns in MySql re-assigned the numbers based on the order in the dump, bypassing gaps created by deleted records. Cross-references via the ID number were all screwed up. It was a pasta nightmare. I'm not sure how to prevent this in the future, other than manually program my own number assignments or program it to ignore records marked deleted instead of actually deleting them.

Power Management configuration

[inKubus]

With newer advances such as Speedstep (or whatever they call it now), you can communicate to the servers that they are in a power outage mode and have them flip into a low power mode. Sure, they will run slower, but they won't go DOWN, and the resulting KW of power not being dissipated in the room will help keep things cooler, longer. Of course, in a colo, this means certain software that the server owner has to install. Better for this to be a hardware integration, with some type of network between the servers and the power management system and the cooling system. I know something like this would be fairly easy to implement, provided you had the right device in each server. I see performance adjustments which can downclock certain processors (laptops have had this for years), turn off hard drives, and blank all the monitors automatically (with an override of course).

Re:Better yet a new power strategy

[evilviper]

alot of the heat in a PS is generated from the transformer in converting the AC power coming to DC used by the computer.

That's flatly incorrect. AC to DC conversion is a very simple and very efficient process.

I will also note that the original post says "device" not "server".

I have no idea what you are talking about. It says what it says, exactly as I quoted it: "device/server"

What's more, no matter how you look at, it's simply NOT true.

A PSU would have to be 49% efficient or less, to output more heat itself than whatever device it is powering (Power==Heat). Meanwhile, even the very cheapest of junk PSUs are 60% efficient. I can't remember when I last saw a PSU that was any less efficient. Meanwhile, it's quite easy and inexpensive to purchase one of many PSUs that are 80% efficient or better, with a few exceeding 90%.

How many servers now have multiple power supplies for redundancy?

I'd say most do, but that's besides the point. No matter how many PSUs you have, they can't possibly be 60% efficient, and yet wasting more power than they are supplying. THAT'S WHAT THE PSU EFFICIENCY NUMBER MEANS.

Even if it is only 15% as you claim, that 15% could stretch 2 hours of run time until critical into 2 hours and 18 minutes.

No it couldn't. That 15% is not something you can get rid of in any way that I know of. It isn't caused by being AC, it's there because it costs some energy to convert from one voltage to another.

If you have some single power supply converting grid power into 48V DC, the 48V DC PSUs that convert into all the standard PC voltages will probably waste 15%+, and that's not including the losses from converting INTO 48V in the first place.

If you go to extremes, and standardize all PC hardware on a single voltage, then the losses will be in every device, and they will easily exceed 15%. Your motherboard will have to convert from a higher voltage down to whatever the CPU, chipset, and RAM needs. And you're still converting AC from the grid into low-voltage DC, which is certainly losing more than 15%, and then some more due to the high current in the electrical lines that now need to be massive. Not to mention that it still won't work, because every time a server turns on or off, the voltage will fluctuate slightly, which will potentially erase you RAM, cause your CPU to output garbage, cause your hard drive to spin down, etc., etc.

This happened where I worked

[LilGuy]

I worked for a mid-sized business ISP a few years ago and we lost our HVACs. All 4 of them at the same time. The maintenance guy that came to do some routine crap was new and somehow completely fubar'd the entire system. We had 2 HVACS for cooling our two data centers and 2 on backup in case anything happened. He somehow managed to shut all 4 down and they absolutely refused to come back online.

It took about an hour before we had to begin shutting the least critical firewalls/servers/routers etc off. We let some pieces of equipment get to the point of melting before we unplugged them, because they were so critical and could be replaced for cheaper at a later time.

We all thought the company was done right then and there. Everyone stayed around the clock trying to think up plans. We had a data center in Minneapolis, 5 hours away that we could run all the big client equipment to, but other than that we were SOL.

Then someone made a few phone calls and managed to procure 8 portable air conditioners. The network engineers cut holes in the wall and ran ducts from the AC units to outside. We all took 2 hour shifts changing the water pans and keeping accurate temperature records throughout the weekend while the HVACs were being replaced.

What a nightmare that was... but we came out of it alright in the end. Didn't lose any major clients and didn't have to shuttle any equipment to minnesota. I bet the guy that got fired for screwin it all up will never work in the hvac industry again though. I still feel kinda bad for him.

power outage -- 'thermal shutdown' ?!?

[frankenheinz]

A recent study found that a data center running at 5 kilowatts per server cabinet may experience a thermal shutdown in as little as three minutes during a power outage. See, now that's interesting. At least, I [erroneously] believed that a power 'outage' might actually prevent a thermal shutdown.

Re:power outage -- 'thermal shutdown' ?!?

[petermgreen]

The problem is when you get an outage of utility power, the servers keep up on first UPS then generator. The aircon goes down and then should come back up when the generators come online but if something goes wrong and the aircon power stays out then there is a problem.

Re:Why Such Density of Servers ?

[petermgreen]

The thing is for better or worse network centers have ended up in city centers, the further you are from those centers the more your connections cost.