The Risks and Rewards of Warmer Data Centers

1sockchuck writes "The risks and rewards of raising the temperature in the data center were debated last week in several new studies based on real-world testing in Silicon Valley facilities. The verdict: companies can indeed save big money on power costs by running warmer. Cisco Systems expects to save $2 million a year by raising the temperature in its San Jose research labs. But nudge the thermostat too high, and the energy savings can evaporate in a flurry of server fan activity. The new studies added some practical guidance on a trend that has become a hot topic as companies focus on rising power bills in the data center."

Comments

Quick solution

[Geoffrey.landis]

Locate the server farm in Antarctica!

Re:Quick solution

[Necroloth]

That'll teach those penguin lovers!

Re:Quick solution

[orsty3001]

If rubbing frozen dirt in my crotch is wrong, I don't want to be right.

Re:Quick solution

[GargamelSpaceman]

It seems to me that computers produce X BTUs of energy that must be taken out of the server room. They will produce this energy regardless of the temperature in the server room. So... with great insulation around the room, the temperature INSIDE the room should not matter much with regards to the cost of keeping it cold. I think you'd want a temperature where the fans never come on at all ideally. How about making the server room a large dewar flask and fill it with liquid nitrogen and running servers? Why should it cost any more to maintain the room at 0 degrees than it would to maintain the room at 100 degrees. I would expect quite the opposite ( with great insulation AROUND the room. )

Re:Quick solution

[snowraver1]

Good thoughts, but I think that cooling equipment works better when the internal temperature is higher. That way the coolant can collect more heat in the evaporation phase, which can then be dumped outside from the condenser. So, by keeping the temperature warmer, the return coolant has more heat in it, but the coolant still evaporates at the same temperature, so you get a larger delta. Yes, the compressers would have to work a little harder, but overall, apperantly it is a net savings in energy.

Re:Quick solution

[autora]

I see you've really thought this one through... A warehouse full of servers that need regular maintenance filled with liquid nitrogen is sure to lower costs.

Re:Quick solution

[jschen]

It is true that if you are producing X BTUs of heat inside the room, then to maintain temperature, you have to pump that much heat out. However, the efficiency of this heat transfer depends on the temperature difference between the inside and the outside. To the extent you want to force air (or any other heat transfer medium) that is already colder than outside to dump energy into air (or other medium) that is warmer, that will cost you energy.

Also, too cold, and you will invite condensation. In your hypothetical scenario, you'd need to run some pretty powerful air conditioning to prevent condensation from forming everywhere.

Re:Quick solution

[fatalwall]

wouldn't you just lose the savings with heating... computers do have a minimum temperature that they will work in...

btw not to be a grammar nazi but its one 'o' not two for lose

Re:Quick solution

[petermgreen]

Just make the building well insulated and then have controlled fans to bring in just enough outside air to keep the temperature where you want it.

Re:Quick solution

[petermgreen]

Antartica would indeed not be a good choice but afaict there are places with temperatures low enough that you could use outside air to cool stuff year round while not being so low as to cause major logistical problems.

Re:Quick solution

[lobiusmoop]

Data centers would be much more efficient if blade servers had modular water cooling instead of fans. Water is much better at transferring heat than air. Then you could just remove all the fans from the data center and add a network of water pipes (alongside the spaghetti of network and power cabling) around the data center. Then just pump cold water in and dispose of the hot water (pretty cheap to do). Should be reasonable safe too really - the water should only be near low-voltage systems really (voltage stepdown should really be happening at a single point in an efficient data center, not at every rack).

Re:Quick solution

[WhatAmIDoingHere]

"You loose your energy savings.."

So all you have to do is tighten those savings and you'll be fine.

Re:Quick solution

[N Monkey]

Locate the server farm in Antarctica!

Perhaps not quite Antarctica, but according to the BBC's Click program Iceland is bidding for server business based on the low temperatures and lots of cheap geothermal power.

Re:Quick solution

[xaxa]

Very high altitude, very cold, very low humidity -- you regularly lose hard drives from head crashes.

Re:Quick solution

[Yetihehe]

Condensation happens on surfaces colder than surrounding air. If you have computers which are warmer than your cooling air, it would not be a problem.

Re:Quick solution

[Mr. Freeman]

"Why should it cost any more to maintain the room at 0 degrees than it would to maintain the room at 100 degrees. I would expect quite the opposite ( with great insulation AROUND the room. )"
Yeah, therein lies the problem. This "great insulation":
A) Doesn't exist.
B) Is horrendously expensive.

Yes, in an ideal environment this makes sense, but we're not working in one. You have energy leak in from the outside. In addition to that, there's no device that can move energy ideally. There's inefficiencies in every thing. Heaters, fans, air conditioning, etc. This causes further energy "losses".

It reminds me of a joke. A farmer goes to a physicist and says "I want to know if a chicken lays an egg on a roof, which way will it roll. Do you have an equation for this". The physicist thinks on this for awhile and then says "I do, but it only works for perfectly spherical chickens in a vacuum".

We have thermodynamic equations to model things like server rooms, but you have two options:
1) Make assumptions about things to simplify the equations.
2) Take into account everything and make your equations so complicated that they become practically unsolvable

Re:Quick solution

[Mr. Freeman]

"That way the coolant can collect more heat in the evaporation phase"
WHAT?!?

The heat of vaporization doesn't change based on temperature. What are you talking about?

Re:Quick solution

[Mr. Freeman]

I thought the reasons for water cooling systems not being placed in data centers were:
1) A failure causing coolant leakage could potentially destroy tens of servers.
2) Maintenance of these systems is quite expensive (mold and such growing int he lines that needs to be periodically cleaned out.)
3) Failure of a main pump could bring down the entire data center (although I assume there would be redundant systems in place)

Re:Quick solution

[Interoperable]

Pffff... /. needs an speling Nazi mod cattegory :-)

Re:Quick solution

[Interoperable]

The "rubbing frozen dirt in my crotch" post get modded "Funny" and I get called out "loose". Not fair ;-)

Re:Quick solution

[billcopc]

You mean like Crays used to have ?

The problems with water are numerous: leaks, evaporation, rust/corrosion, dead/weak pumps, fungus/algae, even just the weight of all that water can cause big problems and complicate room layouts.

Air is easy. A fan is a simple device: it either spins, or it doesn't. A compressor is also rather simple. Having fewer failure modes in a system makes it easier to monitor and maintain.

You also can't just "dispose of the hot water". It's not like you can leave the cold faucet open, and piss the hot water out as waste. Water cooling systems are closed loops. You cool your own water via radiators, which themselves are either passively or actively cooled with fans and peltiers. You could recirculate the hot water through the building and recycle the heat, but for most datacenters you'd still have a huge thermal surplus that needs to be dissipated. Heat doesn't just vanish because you have water, it only allows you to move it faster.

Re:Quick solution

[ElSupreme]

You get better efficiency with trying to cool a warmer enviroment. And you get better efficiency trying to cool with a colder outside temperature. The the smaller the delta T (assuming Cooler inside, warmer outside) the better your efficency.

Re:Quick solution

[treeves]

Try ... offtopic. But you can't anyway since you already posted.

--Part-time grammar/spelling Nazi with no mod points right now.

Re:Quick solution

[DavidTC]

I think we're spending way too much time trying to 'cool' things that do not, in fact, need to be cooler than outside. Nowhere on earth is so hot that servers won't run, unless you've built a server room over an active volcano or something.

All we actually need to do is remove the heat from the servers to the air, and then keep swapping the air with the outside.

Which happens automatically if you let heat out the top and air in the bottom. Even if you have to condition the incoming air to remove moisture, that's cheaper than actually 'cooling' AC. So the second part, replacing the room air, is easy.

As for the first, I've always wondered why they don't use chimney-like devices to generate wind naturally and send it though server racks, instead of fans. I think all the heat in a server room could actually, on exit, suck incoming air in fast enough to cool computers if it actually hit the right places on the way in.

Heck, this would apply anyway. Instead of having AC vent into server rooms, why not have AC vent into server racks? Hook up the damn AC to the fan vent on each server, blow cold air straight in. The room itself could end not cold at all.

Re:Quick solution

[GargamelSpaceman]

Actually I haven't thought much about it at all. For all I know liquid nitrogen is a conductor which would be like filling the server room with ice water. Then again maybe liquid nitrogen is a good insulator.

Re:Quick solution

[GargamelSpaceman]

What about filling the server room with freon, and letting it boil away at 1 atm. Then recompress the freon, radiating the heat away ( the radiator can be arbitrarily hot depending on the power of the compressor providing a very hot thing and as large a temp difference as you could want ) Then use the (cool) liquid freon to top off the server room ( since some has boiled away )?

Re:Quick solution

[lgw]

Allow me to speak for Slashdot grammar nazis and say: we're damn tired of seeing that lose/loose error, in particular. At least with the its/it's and their/there/they're errors, you could see how someone could become confused, but there's just no way anyone would pronounce "loose" as "lose". Seriously, the internet is great and all but ... read a book!

Re:Quick solution

[lgw]

I always figured the best approach was a combined server room/aquarium. But that assumes you can train some fish to do your server maintenance. I hear the octopus is quite smart, and could easily move around inside of cases. I wonder though, will this provoke cries of "fight octopus outsourcing now!" from the Slashdot crowd?

Re:Quick solution

[Interoperable]

I was actually thinking that the spelling/grammar Nazi tag might come with a Karma bonus ;-)

Re:Quick solution

[treeves]

Well, that made me smile. Not what I expected.

Re:Quick solution

[Zerth]

Many racks already do this. Plus, if you aren't looking to pipe AC in each rack, just rotate your racks into alternating hot/cold aisles and seal them off so air can only pass through intakes/outflow vents.

Re:Quick solution

[Interoperable]

we're damn tired of seeing that lose/loose error, in particular

So am I, but I am very familiar with the correct way to spell lose but now and then we all make typos. I don't actually mind being called out on it because it bugs me too, but I don't appreciate the "read a book" comment.

Re:Quick solution

[sjames]

The temperature gradient between the hot and cold side of the cooling system makes a big difference, just like it takes more work to pump 100 gallons of water to a height of 100 feet than it does to pump it 1 foot. Meanwhile if the outside is cooler than the inside the heat will flow with no effort at all.

Re:Quick solution

[mhajicek]

Correction: condensation happens when the temperature drops below the dew point. Dehydrate your air and condensation won't be a problem at any temperature.

Re:Quick solution

[Chris Burke]

Nowhere on earth is so hot that servers won't run, unless you've built a server room over an active volcano or something.

Given a sufficiently powerful fan, then yes.

All we actually need to do is remove the heat from the servers to the air, and then keep swapping the air with the outside.

Which becomes more difficult the higher the ambient air temperature becomes. Heat transfer is proportional to heat delta, so the closer the air temperature is to the heat sink temperature, the more air you need to blow to remove the same amount of heat. Eventually, the amount of electricity you are spending blowing air over the heat sinks is greater than the savings of using less AC.

This was half the point of the article -- you can save a lot of money by raising server room temperatures, but eventually (at a temperature well below outdoor ambient around here) you actually start to lose money due to all the extra fan activity.

Which happens automatically if you let heat out the top and air in the bottom.

Yes but much too slowly to be of use. Convection is also proportional to temperature difference. By the time your server room temperature is enough higher than outside temperature to create significant airflow, your servers are toast.

As for the first, I've always wondered why they don't use chimney-like devices to generate wind naturally and send it though server racks, instead of fans.

Go ahead and try it. A lot of cases already have ducting that funnels air directly from outside the case to the CPU. A few more pieces of cardboard, a hole and chimney in the top of your case, and you should be ready to remove the fan and see what convection can do for you. Sneak preview: unless you've specifically picked components that can run off passive cooling, you'll be in the market for a new one. Especially if you live in a hot place and turn off your AC for this experiment.

While its conceivable to have an effective server room based entirely off of low-power chips that require no active cooling, space is still a major concern in the server room. The desire for greater compute density is directly fighting against using a large number of low-power chips spread out. Thus performance/watt becomes a major metric for the server room, because they want the most performance for a fixed amount of space and thus cooling.

why not have AC vent into server racks?

That's actually a good idea, and a lot of places do it.

Re:Quick solution

[Chris Burke]

I hear the octopus is quite smart, and could easily move around inside of cases. I wonder though, will this provoke cries of "fight octopus outsourcing now!" from the Slashdot crowd?

That depends... are these First World octopi who enjoy all the legal protection of modern democracies? Or are these exploited Chinese sweatshop octopi?

Re:Quick solution

[VolciMaster]

Nowhere on earth is so hot that servers won't run, unless you've built a server room over an active volcano or something.

..and then they wonder why they have LAVA in the LIVING ROOM!

Re:Quick solution

[VolciMaster]

the anti-CFC crowd would have your head for that :)

Re:Quick solution

[SnowZero]

Or are these exploited Chinese sweatshop octopi?

The correct term is saltshop.

Re:Quick solution

[metaforest]

Go ahead and try it. A lot of cases already have ducting that funnels air directly from outside the case to the CPU. A few more pieces of cardboard, a hole and chimney in the top of your case, and you should be ready to remove the fan and see what convection can do for you. Sneak preview: unless you've specifically picked components that can run off passive cooling, you'll be in the market for a new one. Especially if you live in a hot place and turn off your AC for this experiment.

I wanted a REALLY quiet, and powerful server in a small case:
I built a water-cooled Q6600 (Over clocked to 3.65GHz - system dissipates ~550W) into a 2U case, using an external 1KW radiator. I found that using convection tricks did not create enough air flow until the coolant temp was about 40C but then the die-temps on the mobo were too high... The system reached equilibrium (ambient air at 10C -- December with windows open) AT IDLE of about 85C on the dice. That was not workable.

Convection just doesn't cut it, and even if you increase the radiator surface area dramatically, you still have a serious issue: It takes too long for the convection cooling system to respond to changes in thermal load... The dice and coolant will reach an unacceptably high temp long before the convection flow finds a new equilibrium.

My solution was to add two 120mm fans to the radiator. The fans spool based on CPU socket temp. The system holds temp: The die-temp is 32C at idle with an ambient of 25C. Under a 4 thread MPrime load, die-temps hold at 48C.

YMMV

Re:Quick solution

[GargamelSpaceman]

You can put a cup of nothing in liquid nitrogen, collect the liquid oxygen, then pour it on activated charcoal and light it with a long fuse. Supposedly it's a high explosive.

Re:Quick solution

[cthulhu11]

Oh yeah, that "Clever venting" worked out so well for the original 128K Macintosh after all.

Re:Quick solution

[Hurricane78]

The problems with water are numerous: leaks, evaporation, rust/corrosion, dead/weak pumps, fungus/algae, even just the weight of all that water can cause big problems and complicate room layouts.

Eeh, what? I ran a 100% water cooled system (distilled water + 1,2-ethandiol) for 5 years.
- I do not see how the water would evaporate. Out of the tubes?? That takes decades!
- Rust/corrosion? Not with that mix. And we have rust-free materials for ages now.
- Dead/weak pumps? Fans die much quicker, because they have bearings that grind off with time. The normal pumps you use for water cooling have one single moving part, that is hovering in the middle of a magnetic ring, never touching the outside surface. There is no wear of any kind. Only the electronics can die of the same things that your other hardware can die of.
- Fungus/algae? In distilled water with a buttload of 1,2-ethandiol (a mildly toxic substance)?? I don't think so! ^^
- The weight? You mean the one liter that I needed in my system, because of the huge passive cooler and because I cooled everything in the system that way, extrapolated to a server rack, is any relevant to the weight of the whole freakin rack itself?? LOL. The water needed would fit in one or two buckets. You can lift those. But try lifting the filled rack.
- It would not complicate room layouts. You simply have two pipes in parallel to every power cable. Same layout, except for the central water cooler being a bit further down the wall of the hall than the central power supply.Or you simply just use water cooling inside the system, and air cooled heat exchanges outside of it.

The only problems are:
- Exchanging water-cooled parts. (Which can be solved by using proper auto-closing couplings.)
- The non-water-cooled parts better not get any hot. Because there is no air movement to cool them. (Typical problem: voltage converters next to the CPU. Typical solution: Water-cool those parts too.)

You obviously have never used water cooling in computers.

Re:Quick solution

[toddestan]

Go ahead and try it. A lot of cases already have ducting that funnels air directly from outside the case to the CPU. A few more pieces of cardboard, a hole and chimney in the top of your case, and you should be ready to remove the fan and see what convection can do for you. Sneak preview: unless you've specifically picked components that can run off passive cooling, you'll be in the market for a new one. Especially if you live in a hot place and turn off your AC for this experiment.

Actually, Apple tried this with the G4 cube. It more or less worked, but ran awfully hot which tended to make them less reliable than their PowerMac tower cousins. Not to mention the heat caused cracks in the case, which certainly didn't help much for a computer that people bought mostly for its looks.

Apple had better luck with convection on the early iMacs though, using the heat from the CRT to generate enough convection to cool the rest of the computer. The only problem with that design was that the iMac could not shut off the CRT as a power saving feature as it depended upon it for cooling.

Possible strategy

[Nerdposeur]

1. Get a thermostat you can control with a computer
2. Give the computer inputs of temperature and energy use, and output of heating/cooling
3. Write a program to minimize energy use (genetic algorithm?)
4. Profit!!

Possible problem: do we need to factor in some increased wear & tear on the machines for higher temperatures? That would complicate things.

Re:Possible strategy

[jeffmeden]

Careful with that, there are numerous patents to that effect. You wouldn't want to be suggesting IP theft, now, would you?

Re:Possible strategy

[L4t3r4lu5]

Erm... Surely you just replace a room thermostat with a CPU temp probe and the biiiiiiig chillers on the back wall with smaller chillers feeding directly into the cases?

Feedback loop to turn on the chillers above a certain temp and... Bob's your mother's brother.

Re:Possible strategy

[Jurily]

Possible problem: do we need to factor in some increased wear & tear on the machines for higher temperatures? That would complicate things.

And the increased burnout rate of your sysadmins. But who cares about them, right?

Re:Possible strategy

[Monkeedude1212]

Sadly, in an effort to save money, we hired some developers with little to no experience, and zero credentials. Turns out the program they wrote to control the thermostat eats up so many compute cycles that it visibly raises temperature of whatever machine its running on. So we ran it in the server room, because thats where temperature is most important. However by the time it would adjust the temperature the room would raise 1 Degree. Then it would have to redo its analysis and adjustments.

Long story short, the building burned down and I'm now unemployed.

Re:Possible strategy

[Linker3000]

Interestingly enough, I recently submitted an 'Ask Slashdot' (Pending) about this as my IT room is also the building's server room (just one rack and 5 servers) and we normally just keep the windows open during the day and turn on the aircon when we close up for the night, but sometimes we forget and the room's a bit warm when we come in the next day! We could just leave the aircon on all the time but that's not very eco-friendly.

I was asking for advice on USB/LAN-based temp sensors and also USB/LAN-based learning IR transmitters so we could have some code that sensed temperature and then signalled to the aircon to turn on by mimicking the remote control. Google turns up a wide range of kit from bareboard projects to 'professional' HVAC temperature modules costing stupid money so I was wondering if anyone had some practical experience of marrying the two requirements (temp sensor and IR transmitter) with sensibly-priced, off-the-shelf (in the UK) kit.

Anyone?

Re:Possible strategy

[L4t3r4lu5]

No, but I'm visiting Ireland next week! I'll think of you while I'm enjoying a decent beer and looking at some Irish... Well, you know what I mean.

Re:Possible strategy

[dissy]

Careful with that, there are numerous patents to that effect. You wouldn't want to be suggesting IP theft, now, would you?

Of course not! We don't steal IP here. In fact, that sheet of paper with their IP on it (the patent) will forever remain safely tucked away on file at the patent office, safe from all thieves.

We are however suggesting to ignore the fact that patent exists, and use that knowledge anyway.

Even if you want to be anti-capitalist and follow patent law, it is easy enough to use only the methods provided by IBMs expired patents and thus not run a fowl of any laws.

Re:Possible strategy

[Anonymous Coward]

True, but try to run a swine of the law and see what happens.

Re:Possible strategy

[WuphonsReach]

http://www.itwatchdogs.com/products_mon.shtml

Basically, you're looking at $300-$1000 in hardware, but it can interface with Nagios.

If we ever move our servers to the basement, I'll be setting these up to monitor for flooding or temperature issues.

Re:Possible strategy

[Linker3000]

Cheers - my start point for pricing was a USB-based IR transmitter that only costs 67UKP

http://www.redrat.co.uk/products/index.html

Anyone used this for a similar temp control project?

Re:Possible strategy

[JustinRLynn]

Wow, I never thought I'd observe a thermal cascade reaction outside of a chemistry lab or a nuclear power plant. Thanks slashdot!

Re:Possible strategy

[herring0]

We used the Sensatronics EM1 which is connected to the network and monitor it with several things. The EM1 interface is very simple and one of the monitors is just a cron job that scrapes the output from the web interface and will shutdown some of our more sensative equipment if it gets too hot.

They also have a bevy of interfaces from commercial products and the couple of monitoring/notification systems we tested were all able to communicate with the EM1 without any problems.

The total cost for the EM1 and several temp and temp+humidity probes was less than 700$ USD. If you don't care about multiple probes you could probably get it for under 500$ USD.

http://www.sensatronics.com/

Re:Possible strategy

[Nyeerrmm]

You don't need anything as complicated as a genetic algorithm. You have a defined control (thermostat), a defined state (temperature), some external but relatively predictable variables (outside temperature and server load), and a decent model (should be a straightforward ODE) that defines the relationship between those. Define a cost function, balancing the need to keep both temperatures and energy costs low, and you've got a very straightforward optimal control problem.

Because its continuous, not particularly chaotic, and has a limited number of states and controls, basic Lagrangian optimal control methods should be very effective -- they also are little more than solving a set ODEs, and with low precision requirements its computationally easy. GAs are great for very complex problems with discrete components or highly non-linear chaotic components, but are overkill for this kind of problem.

Re:Possible strategy

[PPH]

What? Do you live in some science-fictiony, futuristic world? That sort of thing would be great. But here's how our PHB management deals with such problems. Note that this is an office environment, not a server room:

For years, we've worked in an office that is usually maintained at about 80F, summer or winter. Management mandated that the thermostats all be set to this point so as to save energy consumed by the AC system. During the summer, the math was simple. Its an old building (1950s era), single story, about 6 acres in size (big roof area, not insulated). But during the winter, it was still 80F. That must be one hell of a heat load due to the occupancy, all the desktop PCs running, etc. Probably something not anticipated back when the HVAC systems were designed.

Then the Nisqually earthquake hit (in the Seattle area). The office was shut down for a week. When we finally returned, to our amazement, the building temp was about 65F in the morning, rising to about 72F by quitting time. This was in March. After a few weeks, it went back to 80F.

Some months later, I ran across one of the facilities techs and asked him about the t'stat settings, heat load and why the building seemed to behave differently following the earthquake. He told me that, after that incident, the natural gas supply to the building was shut down for a few weeks until the plumbing could be tested. This disabled the heating plant. 65 to 72 was the natural temp of the office during the winter with normal occupant heat loads. But back when this facility was built, they had installed thermostats with single set points to control both heating and air conditioning. Managements mandate to crank them up to 80F to save on air conditioning costs had resulted in the (uninsulated) building being overheated during cooler weather. So we had been burning money for years (decades?).

Our Type A (A for asshole) management has refused to spend any extra money on 'that damned newfangled technology. The old stuff works just fine.' Nobody has the balls to explain the problem and economics to them. Good luck explaining server room engineering/economics to them.

Re:Possible strategy

[Zerth]

If you are going for cheap, you can get a USB thermometer for like 9 bucks. I'm sure you can find them locally for a bit more.
http://www.dealextreme.com/details.dx/sku.7003

The included software is win-only. Several people have coded some linux tools of varying usefulness. You'll probably want to do your own calibrations, mine is consistantly off, but I've seen others complain of nonlinear responses.

http://err.no/personal/blog/tech/2008-07-22-10-17_kernel_patches_TEMPer_thermometer.html
http://www.roaringpenguin.com/products/temper-tools

As for IR, um...
cheap
not so cheap

Re:Possible strategy

[LifesABeach]

This reminds me of Apple's advertising in 1982 when they were trying to get people to buy a muffin fan for their Applie 2E's, their logo was, "The Only Thing You Can Do With a Baked Apple is Eat It." and a picture below of a partially incinerated Apple 2E.

Re:Possible strategy

[grimsweep]

You win again, Heisenberg!

Re:Possible strategy

[TheLink]

> I was asking for advice on USB/LAN-based temp sensors

But why do you need those sensors? I thought most servers have temperature sensors built in, and they are software readable.

Why not use those instead?

And "show environment" works on some cisco routers.

Even my home pc's motherboard has temp sensors (and so does its cpu).

Re:Possible strategy

[Hurricane78]

You mean something like this?
http://www.webshop-innovatek.de/shop/Fancontroller/Fan-O-Matic-Familie/Fan-O-Matic-PRO---USB-LCD-Display.html

Really a great system by the way. (I'm not affiliated, but I used their parts for years. And they have the best support I have ever seen. You can talk directly to their competent engineers in the forums. Only TerraTec comes close in support friendliness.)

A little bit unclear

[sunking2]

Sure, the fans kick in and you aren't saving as much, but are you still saving? I suspect you still are, there is a reason you are told to run ceiling fans in your house even with the AC on.

The thermal modeling for all this isn't that difficult. You can get power consumption, fan speeds, temp, etc and feed them into a pretty accurate plant model that should be able to on the fly adjust temperature for optimal efficiency. Or I guess we can hire company to form a bunch of committees to do a bunch of studies and come up with a bunch of papers that state the obvious.

Re:A little bit unclear

[mea37]

"Sure, the fans kick in and you aren't saving as much, but are you still saving? I suspect you still are, there is a reason you are told to run ceiling fans in your house even with the AC on."

If only someone would do a study based on real-world testing, we could be sure... Oh, wait...

There are several differences between ceiling fans and server fans. You can't use one to make predications about the other. "Using one large fan to increase airflow in a room is a more efficient way for people to feel cooler than using AC to actually drop the temp a few extra degrees", but this does not imply that "running a bunch of little fans to individually increase heat sink efficiency in each of a number of computers would be moer efficient than just keeping the room cool enough for those heat sinks to do their job in the first place".

Re:A little bit unclear

[greed]

For starters, people sweat and computers do not. So, airflow helps cool people by increasing evaporation, in addition to direct thermal transfer. Even when you think you aren't sweating, your skin is still moist and evaporative cooling still works.

Unless someone invents a CPU swamp cooler, that's just not happening on a computer. You do need airflow to keep the hot air from remaining close to the hot component (this can be convection or forced), but you don't get that extra... let's call it "wind chill" effect that humans feel.

Re:What about HDDs?

[DomNF15]

No they didn't - what they did do is figure out that increased temperature is not correlated to higher failure rates - the failure rates don't magically decrease as it gets hotter.

Here's the link for your review: http://hardware.slashdot.org/story/07/02/18/0420247/Google-Releases-Paper-on-Disk-Reliability

Re:What about HDDs?

[spxero]

[Citation Needed]

Here's a few links to the contrary:
http://www.tomshardware.com/news/google-hard-drives,4347.html
http://tech.blorge.com/Structure:%20/2007/02/20/googles-hard-disk-study-shows-temperature-is-not-as-important-as-once-thought/

Re:What about HDDs?

[Bandman]

Until what point? You can't consistently say "increase the temperature to decrease the MTBF".

You'll end up with molten slag.

The internet's not free?

[commodore64_love]

I thought the internet was free (or so people keep telling me). You mean it actually costs these companies money to maintain the connections??? Wow. I guess my $15/month bill actually serves a purpose after all.

Re:What about HDDs?

[jeffmeden]

Until what point? You can't consistently say "increase the temperature to decrease the MTBF".

You'll end up with molten slag.

Yes, you can. MTBF = mean time before/between failure. To decrease, reduce, lower, however you want to say it, it is going to fail SOONER meaning it is getting LESS reliable. That was the point, hotter temps = less reliability. Same goes for just about any physical/chemical process (fans, batteries, hard drive motors, etc.)

UNITS?

[RiotingPacifist]

80 whats? Obviously they mean 80F (running a temperature at 80K, 80C or 80R would be insane), but you should always specify units (especially if your using some backwards units like Fahrenheit!)

Re:UNITS?

[friedo]

Fahrenheit backwards? That shit was metric before the Metric System even existed.

To wit:

0F is about as cold as it gets, and 100F is about as hot as it gets.

See? Metric.

Re:UNITS?

[ubercam]

lol not around here my friend.

Have you ever gone outside when it's -40 (C or F, it's the same)? The air is so cold that it hurts to breathe, but I love it. There is nothing like it. The humidity from your breath sticks to your eyelashes and they freeze together and you have to pick the ice off so you can open your eyes. It's amazing human beings even live here.

Re:UNITS?

[tepples]

Fahrenheit backwards? That shit was metric before the Metric System even existed.

To wit:

0F is about as cold as it gets, and 100F is about as hot as it gets.

You're right for the 40th parallel or so. But there are parts of the world that routinely dip below 0 deg F (-18 deg C) and other parts that routinely climb above 100 deg F (38 deg C). Things like that are why SI switched from Fahrenheit and Rankine to Celsius and Kelvin.

Re:UNITS?

[nedlohs]

And yet the temperature here measured in F gets negative every winter. And where I previously lived it got above 100F every summer (and it also does where I am now, but only a day or three each year).

But in both those places a temperature of 0C was the freezing point of water, and 100C the boiling point. Yes that 100C one isn't so useful in terms of daily temperature, the 0C is though since whether water will freeze or not is the main transition point in daily temperature.

Re:UNITS?

[jeffmeden]

Things like that are why SI switched from Fahrenheit and Rankine to Celsius and Kelvin.

Sure, blame climate change. Everyone else does!

Re:UNITS?

[Abstrackt]

As a fellow Winnipeger all I can say is this: at least it's not Saskatchewan. =p

Re:UNITS?

[ubercam]

So true! At least we have something mildly interesting to look at, like a tree once in a while, maybe a lake.

Re:UNITS?

[ndege]

in both those places a temperature of 0C was the freezing point of water, and 100C the boiling point

Wow...cool! I have always wanted to live at a location whose conditions matched the International Standard Atmosphere: ie: you lived at sea-level with the temperature at +15 deg C and the pressure at 101,325Pa?

Btw, if it had been said that those values are approximate, I would have let it go. ;)

Re:UNITS?

[DavidTC]

How could the freezing point of water at exactly 0C require that the temperature is 15C? That makes no sense.

Re:UNITS?

[nedlohs]

Both those location were at sea level (well a few feet above given I wasn't always swimming).

And obviously since it's a temperature standard temperature is irrelevant.

You may also notice that I said 0C, I did not say 0.0C or 0.000C. So clearly I was being approximate and yet you didn't let it go likely because you are an idiot.

Re:UNITS?

[Chris Burke]

since whether water will freeze or not is the main transition point in daily temperature.

Surely you meant whether carbon dioxide will freeze or not. Wait... what planet is this site hosted on?

Re:UNITS?

[metaforest]

0F is the freezing point of salty water (ocean water) this is the reference point.

Not clear from what I have found what the step reference is...

Re:UNITS?

[ndege]

and yet you didn't let it go likely because you are an idiot

lol! :) I feel like I am in 3rd grade again.

Anyway, sorry, I didn't originally intend to sound rude, mainly just wanted to point out that life isn't as precise or concrete as many people like to think it is. ie: water doesn't _always_ boil at exactly 100C. Even +/-1C

To restate differently: The idea is that one must make assumptions; in this case, one must assume standard conditions such as density altitude before we can say that water will boil at 100C or freeze at 0C. (As an aside, I have experienced supercooled water while on a camping trip; it was very weird to disturb the water and see the ice crystals form very rapidly where a moment before the water was liquid.) ....but, none the less, thanks for pointing out my idiocy! Ya put a smile on my face. ;)

Re:UNITS?

[nedlohs]

What's the point of insulting people if they smile?

Tears of self disgust and shame, that's the ticket.

The occasional suicide perhaps...

Re:UNITS?

[toddestan]

I've always considered anything below 0F to be damn cold, and anything above 100F to be damn hot. You could also consider them dangerously cold and dangerously hot, as that's about where you can really start to get into big trouble if you venture out into those temperatures and you are not prepared for it.

The biggest problem I've had with Celsius is that 100C really doesn't mean anything to me on a day to day basis. About 55-60C is about the maximum, which is "can touch momentarily but can't hold your finger against it". Anything greater than that is simply too hot to touch.

Re:UNITS?

[tepples]

The biggest problem I've had with Celsius is that 100C really doesn't mean anything to me on a day to day basis.

Other than "the point at which the kettle whistles" or "the point at which the pasta gets dumped into the pot"?

Ducted cabinets

[tom17]

So what about having ductwork as another utility that is brought to each individual server? Rather than having thousands of tiny inefficient fans whirring away, you could have a redundant farm of large efficient fans that pull in cool air from outside (cooling only required then in hot climates or summer) and duct it under the floor in individual efficient ducts to each cabinet.Each cabinet would then have integral duct-work that would connect to individual servers. The servers would then have integral duct-work that would route the air to the critical components. There would have to be a similar system of return-air duct-work that would ultimately route back to another redundant farm of large efficient fans that scavenge the heated air and dump it outside.

I realise that this is not something that could be done quickly, it would require co-operation from all major vendors and then only if it would actually end up being more efficient overall. There would be lots of hurdles to overcome too... Efficient ducting (no jagged edges or corners like int domestic HVAC ductwork), no leaks, easy interconnects, space requirements, rerouting away from inactive equipment etc etc etc.You would still need some ac in the room as there is bound to be heat leakage from the duct-work, as well as heat given off from less critical components, but the level of cooling required would be much less if the bulk of the heat was ducted straight outside.

So I know the implementation of something like this would be monumental, requiring redesigning of servers, racks, cabinets and general DC layout. It would probably require standards to be laid out so that any server will work in any cab etc (like current rackmount equipment is fairly universally compatible), but after this conversion, could it be more efficient and pay off in the long run?

Just thinking out loud.

Tom...

Re:Ducted cabinets

[Cerberus7]

THIS. I was going to post the same thing, but you beat me to it! APC makes exactly what you're talking about. They call it "InfraStruXure." Yeah, I know... Anywho, here's a link to their page for this stuff.

Re:Ducted cabinets

[Linker3000]

While I was looking at aircon stuff for our small room, I came across a company that sold floor-to-ceiling panels and door units that allowed you to 'box in' your racks and then divert your aircon into the construction rather than cooling the whole room. Seems like a sensible solution for smaller data centres or IT rooms with 1 or 2 racks in the corner of an otherwise normal office.

Re:Ducted cabinets

[LMacG]

I know just the man to work on this -- Archebald 'Harry' Tuttle.

Re:Ducted cabinets

[0100010001010011]

Why even have individual cases? It seems to be rare now days that a full rack isn't just full of computers. Why not have one massive door and a bunch of naked computers on the racks. Set up the air flow in your building such that one side is high pressure the other side is low and blow air across the entire thing.

Re:Ducted cabinets

[jeffmeden]

Ah, erm, no. That's not what InfraStruXure is. And there is a good reason. What happens when you need to work in the front or back of the cabinet? All of a sudden your cooling mechanism is offline and you have precious few minutes without forced air before your servers roach themselves.

The reason this has never (and probably will never) been done is the amount of form factor standardization required from top to bottom in the vendor lineup. Even if the heavens parted and God himself handed down a standard, it wouldn't be six months before some vendor decided they would one-up the market and come out with something "Better" that breaks it, and then you are back to the basics. Given the huge amount of heterogeneity in a given data center, asking every bit of equipment to conform to the same standard, and to stick to that standard for more than one product release cycle, is something of a pipe dream.

Re:Ducted cabinets

[turbidostato]

"asking every bit of equipment to conform to the same standard, and to stick to that standard for more than one product release cycle, is something of a pipe dream."

Yeah... I dream the day they decide, well, I don't know, something like all server-grade equipment to fit into a cabinet 482.6mm wide and heigth by multiples of 44.45mm

Re:Ducted cabinets

[joib]

Cray has something like this in the XT series. Each cabinet has a single big fan at the bottom, sucking in air from below the raised floor and pushing it vertically through the system. And since there are no hard drives either, that fan is the only moving part in the entire cabinet.

I think newer versions also have liquid cooling in addition to the big fan. Basically something like a car radiator at the top of the cabinet.

Re:Ducted cabinets

[Leolo]

One huge problem with your proposition is that everyone is moving away from specialised hardware to commodity components. What you are proposing is a return to specialisation.

Re:Ducted cabinets

[Shawndeisi]

I have a few problems with this strategy...

You can already use blade servers/chassis which have two fans on IBM BladeCenter H models, and a larger number of smaller fans (8) on the HP C7000 blade chassis (incidentally I believe the C7000 is more efficient than IBM H chassis on average, but it's been a while since I've specced full chassis out). You can jam 14/16 blades (hp/ibm respectively) into 4 chassis per rack. The servers have an extremely dense footprint and the newer blades Nehalem procs will virtualize the vast majority of workloads. Put these chassis into a datacenter with a pressurized cold row (still use existing cooling and raised floor ideas, just put a door preventing cold row from mixing with hot row) and all of the cooled air will have to go through the server and the fan can spin at lower RPMs to cool, but is still there in case of adverse conditions. You can build extremely efficient racks this way without need for all of the cooperation between vendors that your solution requires. Lastly, for all of the workloads that you can't virtualize with bladecenters and external storage, you can get large boxen that have large fans in them anyhow, and you wouldn't need too many of these.

I just don't see the need to make the solution particularly complex when there are already very viable and dependable/reliable strategies to deal with this.

Cluster load balancing based on temp

[EmagGeek]

Well, if you have a large cluster, you can load balance based on CPU temp to maintain a uniform junction temp across the cluster. Then all you need to do is maintain just enough A/C to keep the CPU cooling fans running slow (so there is excess cooling capacity to handle a load spike since the A/C can only change the temp of the room so quickly)

Or, you can just bury your data center in the antarctic ice and melt some polar ice cap directly.

We have replaced..

[UncleWilly]

We have replaced Tom's Decaf with DOUBLE ESPRESSO this morning, let's see if he's noticed the difference..

Re:What about HDDs?

[MrNemesis]

Except the google study didn't display any evidence of this happening - there was no correlation between higher temperatures and higher failure rates on mechanical drives.

http://hardware.slashdot.org/article.pl?sid=07/02/18/0420247
http://www.engadget.com/2007/02/18/massive-google-hard-drive-survey-turns-up-very-interesting-thing/
http://labs.google.com/papers/disk_failures.pdf

Even if it were, it'd be easy to rememdy - boot all your servers off SSD and keep them in a "hot" room. Keep your SANs-full-o'-spinning-rust in a "cold" room. You've just saved a fortune in air con despite being unable to convince your CTO that heat isn't as big a killer as many people claim it to be.

We had a power failure at one of our data centres, due to a combination of a stupid JCB driver and IBM's ineptitude (not keeping the diesel tanks full). Power for the servers was restored about six hours before the air-con was back up and running, and most of our equipment got cooked (ambient temp ranging from 35 to 40 degrees depending which part of the data centre you stood in) - we demanded IBM guarantee us a 3hr turnaround on any parts that died for the next 6 months due to heat failure. 18 months later and our hard drive failure graph is the same as it ever was.

Shoddy components on hardware is another matter I guess, but we've never had any hardware die due to a single faulty component apart from the occasional RAID card. Expect, as "hot" DC's become more common, that the heat thresholds on lots of enterprise equipment will increase... for a price, of course ;)

Turn fans down?

[RiotingPacifist]

I used to have a Pentium 4 Prescott , the truth is processors can run significantly above spec (hell the thing would go above the "max temp" just opening notepad). It's already been shown that higher temps don't break HDD, are the downsides of running the processor a few degrees hotter significant or can they be ignored?

Re:Turn fans down?

[TheRaven64]

With the P4, there are significant disadvantages with running it too hot. The CPU contains a thermal sensor and automatically reduces the clock speed when it gets hot to prevent damage. When it's running at 100MHz, it takes a while to get anything done...

Move to Canada

[Midnight Thunder]

I know it was meant as a joke, but moving to colder climates may not be such a bad idea. Moving to a northern country such as Canada or Norway, you would benefit from the colder outside temperature, in the winter, to keep the servers cool and then any heat produced could be funnelled to keeping nearby buildings warm. The real challenge will be keeping any humidity out, but considering how dry the air during the winters can get there it may not be any issue.

All this said and done, trying to work out the sweet spot between not cooling a room to save energy and not having the server fans turn on is important. I would be curious to know if there are any solutions that allow the system temperature monitors to be linked into a central system, which is then linked to the room's climate control system exist?

Re:Move to Canada

[asdf7890]

I know it was meant as a joke, but moving to colder climates may not be such a bad idea. Moving to a northern country such as Canada or Norway, you would benefit from the colder outside temperature, in the winter, to keep the servers cool and then any heat produced could be funnelled to keeping nearby buildings warm.

There has been a fair bit of talk about building so-call "green" DCs in Iceland, where the lower overall temperatures reduce the need for cooling (meaning less energy used, lowering operational costs) and there is good potential for powering the things mainly with power obtained from geothermal sources.

There was also a study (I think it came out of Google) suggesting that load balancing over an international network, like Google's app engine or similar, be arranged so that when there is enough slack to make a difference more load is passed the DCs that are experiencing more wintery conditions than the others. It makes sense for applications where the extra latency of the server perhaps being the other side of the world some of the time isn't gonig to make much difference to the users.

Re:Move to Canada

[smellsofbikes]

That was my first thought a year ago, when Iceland was going bankrupt: Google should buy the whole place. They have nearly free power because of all the hydroelectric, the ambient temperature is low, they have gobs of smart engineering and IT people looking for work, and Icelandic women are really hot.

Re:Move to Canada

[speculatrix]

when Iceland was going bankrupt: Google should buy the whole place....Icelandic women are really hot

Ah, that's why you never see Icelandic women working in data centres, they overload the air-con!!!

Re:Move to Canada

[ffejie]

You mean something like this? Cisco Building Mediator.

Re:Move to Canada

[imhennessy]

The temperature difference also improves the efficiency for energy reclamation systems. Whether you end up mimicking geothermal on a small scale, or put turbines in exhaust stacks, the sustained savings over the lifetime of the DC should be considerable.

ivan

Hardware upgrades probably nullify the problems

[Yvan256]

If you save enery by having warmer data centers, but that it shortens the MTBF, is it really that big of a deal?

Let's say the hardware is rated for five years. Let's say that running it hotter than the recommended specifications shortens that to three years.

But in three years, new and more efficient hardware will probably replace it anyway because it will require, let's say, 150 watts instead of 200 watts, so the old hardware would get replaced anyway because the new hardware will cost less to run in those lost two years.

Re:Hardware upgrades probably nullify the problems

[amorsen]

But in three years, new and more efficient hardware will probably replace it anyway because it will require, let's say, 150 watts instead of 200 watts

That tends to be hard to get actually, at least if we're talking rack-mountable and if you want it from major vendors.

Rather you get something 4 times as powerful which still uses 200W. If you can then virtualize 4 of the old servers onto one of the new, you have won big. If you can't, you haven't won anything.

Longer Study

[amclay]

The studies were not long enough to constitute a very in-depth analysis. It would have to be a multi-month, or up to a year to analyze all the effects of raising temperatures.

For example, little was considered with:

1) Mechanical Part wear (increased fan wear, component wear due to heat)

2) Employee discomfort (80 degree server room?)

3) Part failure*

*If existing cooling solutions had issues, it would be a shorter time between the issue and additional problems since you have cut your window by ~15 degrees.

Re:Longer Study

[gx5000]

This sounds like Newbies at play... Keep it cool and it will last.. Let it go warm, and hardware sales will wipe out your savings.... Then there the time our power went out and the backup to the building was wiped out by MetroPower, The server room became a sweat lodge with as many fans as we could find. Had we been warm in there we would have lost numerous server towers and Raid Racks... I like my server room served at 69 F / 19 C ... chilled not baken TY.

Re:Longer Study

[LWATCDR]

"Had we been warm in there we would have lost numerous server towers and Raid Racks."
Don't your servers have thermal protection?
Don't you have a power down temp and procedure?
I can not imagine a professional allowing hardware to fry just because they loose AC. You must have a procedure for that situation because it will happen if you run a data center long enough.

Re:Longer Study

[Anonymous Coward]

Don't you have a power down temp and procedure?

Funny story about a time I was in a similar situation.

With shutdown at 90F, our procedure, made up on the fly when we hit 85F and started getting nervous, was to open the doors the machine room, and carry large cardboard sheets in and out of the room in a circle, walking through it and circulating the air like bees do at the front of a hive on a hot day.

It got the room down to 80F in the hour it took for the big-ass blower fans to get there, and the fans kept it stable in the high 70s for the rest of the day. Zero downtime.

The HVAC was fine; the root cause was that building maintenance shut off the water to the AC units without notice to us, the tenant. Yeah, we chewed them out royally the next day, but the first priority was to keep the machines running.

So in a roundabout way, we had a procedure. It was just pretty absurd. On the other hand, we didn't care how dorky we looked; it worked!

Re:Longer Study

[LWATCDR]

Your lucky, here the outside air temp is often higher than your shutdown temp.
  The thing is that having a higher standard temp in the server room will not mean that your server "fries" when you the HVAC goes down. It means that you may have to take down the machines. When I worked in a place with a big machine room our priorities where protect the data, protect the machine, uptime. Our shutdown was 87 and we got to 85 once during a back up power test that failed to power the HVAC.

Re:Longer Study

[Leolo]

Has a study like this been done for the current system of cold server rooms?

1- Mechanical wear (colder = looser fit);
2- Employee discomfort;
3- Part failure*

* In a large server center, I doubt those 15 degrees difference would gain you much in reaction time.

Re:What about HDDs?

[jeffmeden]

After giving that paper a closer look (the best link is this one, btw, the engaget link is dead: http://research.google.com/archive/disk_failures.pdf ) The failure rate went up with cold AND hot temperatures. How they got the disk temps that cold is beyond me, but their hot end seemed a little optimistic since I have seen desktops in comfortably air conditioned rooms running disk temps of 50C or more, and have a strong set of anecdotal evidence that these are the disks that fail most often.

Time to recover

[afidel]

Yes, but if you have the room at the tipping point what does this do to your ability to recover from a fault? I know one reason many datacenters have experienced outages even with redundant systems is that the AC equipment is almost never on UPS and so it takes some time for them to recover after switching to generators. If you are running 10F hotter doesn't that mean you have that much less time for the AC to recover before you start experiencing problems? For a large company with redundant datacenters or in Cisco's case where they are mostly development labs it probably is worth the risk, but for your average small to midsized corporate datacenter it's probably smarter to stay with the tried and true.

Newer tech may reduce temperature issues

[Anonymous Coward]

The use of SSDs in data centers can dramatically impact power usage and temperature management costs:

"The power savings for the SSD-based systems is about 50 percent, and the overall cooling savings are 80 percent, according to the white paper. These savings are significant for a datacenter that spends 40 percent of its budget on power and cooling, and they're bound to make other datacenter operators sit up and take notice." http://arstechnica.com/business/news/2009/10/latest-migrations-show-ssd-is-ready-for-some-datacenters.ars

While MTBF and unit cost are still concerns, the potential savings will likely see more centers moving in this direction.

Another server room horror story

[BenEnglishAtHome]

I'm less concerned with the fine-tuning of the environment for servers than I am with getting the basics right. How many bad server room implementations have you seen?

I'm sitting in one. We used to have a half-dozen built-for-the-purpose Liebert units scattered around the periphery of the room. The space was properly designed and the hardware maintained whatever temp and humidity we chose to set. They were expensive to run and maintain but they did their job and did it right.

About seven years ago, the bean-counting powers-that-be pronounced them "too expensive" and had them ripped out. The replacement central system pumps cold air under the raised floor from one central point. Theoretically, it could work. In practice, it was too humid in here the first day.

And the first week, month, and year. We complained. We did simple things to demonstate to upper management and building management that it was too humid in here, things like storing a box of envelopes in the middle of the room for a week and showing management that they had sealed themselves due to excessive humidity.

We were, in every case, rebuffed.

A few weeks ago, a contractor working on phone lines under the floor complained about the mold. *HE* got listened to. Preliminary studies show both penicillin (relatively harmless) and black (not so harmless) mold in high concentrations. Lift a floor tile near the air input and there's a nice thick coat of fluffy, fuzzy mold on everything. There's mold behind the sheetrock that sometimes bleeds through when the walls sweat. They brought in dehumidifiers that are pulling more than 30 gallons of water out of the air every day. The incoming air, depending on who's doing the measuring, is at 75% to 90% humidity. According to the first independent tester who came in, "Essentially, it's raining" under our floor at the intake.

And the areas where condensation is *supposed* to happen and drain away? Those areas are bone dry.

IOW, our whole system was designed and installed without our input and over our objections by idiots who had no idea what they were doing.

So, my fellow server room denizens, please keep this in mind - When people (especially management types) show up with studies that support the view that the way the environment is controlled in your server room can be altered to save money, be afraid. Be very afraid. It doesn't matter how good the basic research is or how artfully it could be employed to save money without causing problems, by the time the PHBs get ahold of it, it'll be perverted into an excuse to totally screw things up.

Re:Another server room horror story

[infinite9]

About a year ago, I worked on a project in a backwards location that was unfortunately within driving distance of the major city where I live. The rate was good though so I took the job. These people were dumb for a lot of reasons. (it takes a lot for me to call my customers dumb) But the one that really made me laugh was the server rack strategically placed in the server room so that the server room door would smack into it whenever someone came in the room.

Re:Another server room horror story

[bill_kress]

You left out what is usually the best part!

For his valiant efforts in preventing waste did the bean counter get promoted to VP level or directly to an officer of the company? Or did he quit (get pushed out) and get a higher paying job elsewhere. This kind of stupidity never goes unrewarded.

Re:Another server room horror story

[Valdez]

studies show both penicillin...

Apparently you didn't read your new health insurance policy. The bean counter was killing two birds with one stone... the penicillin growth was part of his plan to cut employee health benefits. You don't even have to visit the doctor to get your dose!

Re:Another server room horror story

[BenEnglishAtHome]

We're going to "re-stack" our server room soon. One of the biggest fights we had was making double-damn sure that the double doors to the server room could be fully opened, locked back, and a pallet and pallet jack rolled into the room. Then we need to be able to close the doors behind the jack and rotate the pallet in any direction. This means we need a few square feet inside the doors to be kept completely clear of walls, equipment, fencing, etc. I couldn't believe the crap we caught over this one. The people who draw floorplans for us thought we were crazy to want to waste that much space. We've prevailed for now. Who knows what the flooplans will look like by the time the work crews arrive?

Re:Another server room horror story

[Linker3000]

One of my customers had their on-site facilities guys build them a small computer room from a section of open office space. The guys fixed it so the lights in the new room operated independently of the rest of the floor using a pull cord - but they also managed to put the 'big red off button' on the wall behind the cord so it was a common occurrence for someone to open the door, grope for the light cord, hit the red button and power off the room! The fix was a perspex dome over the red button.

On another site, the whole building was new from scratch, but the architect (or builders) got the plans wrong and the under-floor void in the computer room was 4 feet deep instead of 4 inches - you can imagine the room with no floor fitted; the door was half way up the wall! The fitting company had to have extra-long floor columns made to hold up the floor panels and you had to be careful not to disappear into the void and break your neck in the process if someone had a panel lifted! Bonus - room under the floor for some additional server gear, but you needed a flashlight to go work on it.

Final one: I had a computer room built to house a large Vax 11/750, a broadcast editing suite and some specialist, large format film processing gear - we had no end of fun here...

1) The electrician fitted the room stat on a pillar near the windows and there was enough heat radiation hitting it on sunny days to convince the stat that the room was still warm so the aircon used to deep freeze the room - and vice, versa on cold days! We had the stat moved

2) Early in the build, we couldn't actually keep the room cool enough and we had a stream of aircon technicians looking at the HVAC gear, replacing things, calculating heat loads etc. and scratching their heads. Turned out that the room builders had sealed the radiators into the new dry-lining walls - but hadn't actually turned them off so we had hot-panel walls!

Warmer, better, faster...

[pckl300]

I was at a Google presentation on this last night. If I remember correctly, I believe they found the 'ideal' temperature for running server hardware without decreasing lifespan to be about 45 C.

Re:Warmer, better, faster...

[jbengt]

Are you sure they were'nt measuring this temperature somewhere inside the server rack rather than the ambient temperature in the room? 45C is around 113F, which is hotter than most daytime highs in Phoenix, Arizona, and typical motors are only rated for ambient conditions of 104F, i.e. 40C (not sure about the little motors inside drives, etc.). See Skapare's post below about the failures he experienced.

Re:Warmer, better, faster...

[pckl300]

Yes, sorry. Should have made that clear. They weren't talking about ambient temp. I believe it was in the rack.

Re:What about HDDs?

[petermgreen]

The google survey appears to show that drives are happiest arround 35-40 celcius with failure rates increasing both sides of that band.

Of course there are a couple of issues with that data

Firstly the data comes from the drives built in sensors so if a particular brand of drive has both an abnormal failure rate and an abnormal reported running temperature (either due to producing a different ammount of heat or due to a bad sensor) it would skew the results.

The second problem is they simply don't have much data outside the range 25-45 celcius.

Still I don't think it's a huge issue as long as you don't try to run your datacenter insanely hot.

Risk of AC failure

[Skapare]

If there is a failure of AC ... that is, either Air Conditioning OR Alternating Current, you can see a rapid rise in temperature. With all the systems powered off, the latent heat inside the equipment, which is much higher than the room temperature, emerges and raises the room temperature rapidly. And if the equipment is still powered (via UPS when the power fails), the rise is much faster.

In a large data center I once worked at, with 8 mainframes and 1800 servers, power to the entire building failed after several ups and downs in the first minute. The power company was able to tell us within 20 minutes that it looked like a "several hours" outage. We didn't have the UPS capacity for that long, so we started a massive shutdown. Fortunately it was all automated and the last servers finished their current jobs and powered off in another 20 minutes. In that 40 minutes, the server room, normally kept around 17C, was up to a whopping 33C. And even with everything powered off, it peaked at 38C after another 20 minutes. If it weren't so dark in there I think some people would have been starting a sauna.

We had about 40 hard drive failures and 12 power supply failures coming back up that evening. And one of the mainframes had some issues.

Re:Risk of AC failure

[rbcd]

> We had about 40 hard drive failures and 12 power supply failures coming back up that evening.

That could have just been due to an infrequent shut down. Hard drives are known for not being able to spin back up after being run for a very long time, for example.

Re:Risk of AC failure

[zerocool^]

Generators, man.

You need to invest in power switching equipment and generators; probably a couple of 1MWatt generators. They're big 16 cyl. diesel engines sitting on top of multi-hundred gallon fuel tanks that can be refueled while in operation.

Your UPS cluster should be able to take the load of the building for 20-30 seconds, long enough for the generator to spin up for load. Good generators can spin up in 5-7 seconds.

Good Grief, I can't imagine a UPS cluster capable of sustaining a datacenter for 40 minutes. Not that 16 cyl diesel engines are that eco-friendly, but think of all the lead-acid and bad chemicals in 40 minutes x 1MW of UPS?

Re:Risk of AC failure

[TheLink]

Uh you miss the point.

The problem was not that his servers didn't have power for long enough.

His problem was he had no cooling.

Generators to keep your cooling systems working can cost a fair bit more.

Re:What about HDDs?

[Kashgarinn]

I'm pretty sure if they did a check between vibration and failure rates, they'd get correlation.

I'll also put out there a hypothesis that the rack which has no moving parts will have less failure rates compared to the rack which has moving parts.

Wouldn't it be fun to be a head engineer at one of the bigger companies and be able to test it out :)

keep UPS separate and cooler

[speculatrix]

UPS batteries are sealed lead-acid and they definitely benefit from being kept cooler, it's also good to keep them in a separate room, usually close to your main power switching. As far as servers are concerned, I've always been happy with ab ambient room temp of about 22 or 23, provided air-flow is good so you don't get hot-spots, and it makes for a more pleasant working environment (although with remote management I generally don't need to actually work in them for long periods of time).

Re:keep UPS separate and cooler

[jbengt]

You are deefinitely right about the UPS benefitting from cooler air, but I have been involved in several projects where A/C was not installed for the UPS rooms, just ventilation; and this in a Chicago climate where the summer temperature often exceeds 95F (35C). The systems ran fine, though we did let the owner know that they were shortening the life of the system.
As far as 22C or 23C goes, that is a cooler than typically req'd now; 75F to 78F (24C to 27C) is usually perfectly fine for UPS rooms or even server rooms.
It is the phone equipment and other odd pieces that commonly drive the temperatures down and humidities up. They often have specs requiring colder conditions and closer tolerances for humidity and temperature than the bog standard servers. Though I doubt that they really require such, I'm not going to argue with product data published by the manufacturer.
Depending on how large your system is, it's not only better to keep the batteries in a separate room, it's mandated by many codes.

Be very afraid of a study that ...

[Skapare]

describes temperatures using the Fahrenheit scale.

Re:Be very afraid of a study that ...

[Ksevio]

Why? Because that suggests it's from the US? Or that it's aimed at people in the US?

Re:What about HDDs?

[wsanders]

I am a little skeptical since most hard drive failures I've had have been right after a air conditioning outage. The Google paper uses temperature obtained from SMART, which is usually 10 to 15C higher than the ambient temperature in the room, and the tail of their sample falls off rapidly over 40C. What would the SMART temperature be if the ambient temperature was 40 or so? Probably 60 or above. Their graphs don't do that high.

But we're talking raising the temperature of a data center only 2 or 3 deg. Meat lockers are not helpful. Moral of the story? Maybe spend your cooling bucks on your storage, then let the rest of your systems eat their exhaust. I have some new Juniper routers, no moving parts inside except fans - the yellow alarm doesn't kick off until 70C and the machine doesn't shut down until 85C.

Re:What about HDDs?

[Mr. Freeman]

"Wouldn't it be fun to be a head engineer at one of the bigger companies and be able to test it out :)"
Oh really?

Let's see your proposal, your test criteria, your plan.
Let's see your budget... cut it in half
Now for risk analysis, what if you're right and the servers all fail sooner than expected (i.e. sooner than budgeted)?
Spend 3 weeks filling out red tape
Spend 2 weeks waiting.

OK, you can run your study. Set up two racks in a closet and take measurements every day for a year.

Now write up the review.

Alright, thanks for your study, but our lawyers have advised us that it wasn't peer reviewed and published in a respected compsci journal and therefore we can't do anything with it, or the insurance wouldn't cover us and we'd be liable for deaths resulting from servers or something.

File in circular file or far back of filing cabinet never to be seen again until you're clearing out your office because they had to let you go because server replacement costs were too high to keep you on the payroll.

Re:Possible strategy, dude this is /.!

[q2a]

This is slashdot, OF COURSE you should use Nagios!

And to increase your /. Kung-Fu, buy an EM01;
http://www.nagios.org/products/environmental

Learn Nagios the FAN way;
http://fannagioscd.sourceforge.net/drupal/

or play with GroundWork, they're awesome;
http://www.groundworkopensource.com/community/community-edition.html

(Yes, I actually run this in a real data center, we eat our own dog food.)

Lusers [Re:Quick solution]

[Geoffrey.landis]

we're damn tired of seeing that lose/loose error, in particular

Just spell it "luse", and everybody wins.

Sigh...

[BenEnglishAtHome]

Yes, he got promoted to a higher level of management. At least he's now one step further removed from the actual facilities he manages and can no longer screw things up quite as directly as he did in the past.