Slashdot Mirror
Cooler Servers or Cooler Rooms?
mstansberry writes "Analysts, experts and engineers rumble over which is more important in curbing server heat issues; cooler rooms or cooler servers. And who will be the first vendor to bring water back into the data center?"
Unless you make things so cold as to prevent things from working properly, why not just do both?
DBA? Software Engineer? My company is hiring! Click
Will probably be the first vendor to bring water into the datacenter... I believe I've seen evidence in some datacenters already.
it sounds like they're having some kind of gang warfare over the topic...what the hell?
Le français vous intéresse?
I've always wondered this. why have duplication of a function in a server across every single server box when it could all be done in the environment. For example all servers get electricity from the server room and all servers get network from the server room so why not all servers get cooling from 10F cooling in the server room.
It makes sense!
I like cooler rooms. Especially for a large number of servers. Its more efficient.
"We Don't Need No Truthless Heros!" - Project 86
So take your pick. To make the servers cooler, either buy new more efficient servers or buy a whacking great air con unit.
Since the servers are the things that actually do the work, I'd just get feck off servers and a feck off air-con unit to keep it all happy!
Everyones a winner!
Unlike most companies that are considering going back to water cooling, Cray has always used water cooling for their big iron. In fact, the only air cooled Crays are the lower end or smaller configured systems.
All hail the Cray X1E !
Lots of A/C. (more where that came from).
Dashboard Widgets
I agree, both solutions would help. Our room is a nice cool 62.5. Best condtions to work in!
Cooler rooms also keep others out... we get a lot of, its so cold, and they leave. That's golden =)
"I cannot think of any need in childhood as strong as the need for a father's protection." -- Sigmund Freud
...you won't need as much cooling in the room. Easy enough. This will save a ton of money in the long run, not to mention the environment and all that.
That might keep the odd CPU or two cool for a while...
"Roger Schmidt, chief thermodynamics engineer at IBM, [recently] admitted that, while everyone knows servers are one day going to be water-cooled, no one wants to be first, believing that if their competitors still claim they are fine with air cooling, the guy who goes to water cooling will rapidly drop back in sales until others admit it is necessary."
you know, some times the market actually rewards innovation. tough to believe, i know, and this isn't innovation, it's common sense, but mfg's are afraid of this? come on, people, the technocenti have been doing this for their home servers for a long, long time, let's bring it into the corporate world.
nothing worth possessing isn't possessed. or something.
Ideally, you should have a cool server and and cool room. The two work in combination. If you have a hot room, then the server isn't going to be able to cool itself very well even with the best heatsinks and well-placed fans. Yes, you could use water cooling, but there are other important bits inside of a machine besides whatever the water touches. But a cool room does no good if your servers aren't setup with proper cooling themselves.
That comes to mind is that it will probably be vastly cheaper to cool a rackmount specifically than to lower the ambient temperature of an entire room to the point that it has the same effect. However, I'm not entirely sure how well this scales to large server farms and multiple rackmounts.
I think the best option would be to look at having the hardware produce less heat in the first place. This would definitely simplify the rumbling these engineers are engaged in.
In C++, friends can touch each others private parts.
Water cooling? Pah! Why not take a leaf out of Seymour Cray's book - build a sodding great swimming pool, fill it with non-conductive freon, then just lob the whole computer in.
Also has the added benefit that you can see at a glance which processors are working the hardest by looking to see which are producing the most bubbles.
Wonder if you could introduce fish into the tank and make a feature of it? If you could find any freon-breathing fish, that is...
that you should stop the problem where it starts. Cool the servers, then the room won't get hot(duh).
Maybe my ignorance is showing here, but does any installation use outside air for cooling? It seems that it would make sense in places that have cold winters (like here in the midwest).
You'd need a lot of filtering and/or humidity control to make that a realistic option. Better yet to make use of outside air temperature. Which is exactly what your heatpump loop or your AC cooling tower is for.
The sign on the door clearly states, "No Food or Drink". Of course, shirts are still optional.
So long, michael. Don't let the door hit you...
We used to do exactly this in an ops building belonging to the company I worked for in 1997. The server room was cut off from the rest of the heating system in the building, with piped cold air from outside.
It took 8 months until the first servers started dying from the intense corrosion & pitting on equipment closest to the air outlets. We were bringing in air that while it was ice cold, was unfiltered and brought pollution in from 2 storeys above street level, and I dare say more moisture than the air conditioned recycled worker-breathable air.
Filters fixed the problem.
From experience of aircon failing/breaking.
At least if a server fails it's one unit that'll either get hot or shutdown which means a high degree of business continuity.
When your aircon goes down you're in a whole world or hurt. Ours went in a powercut, yet the servers stayed on because of the UPSes - hence the room temperature rose and the alarms went off. Nothing damaged, but it made us realise that it's important to have both, otherwise your redundancy and failover plans expand into the world of facilities and operations, rather than staying within the IT realm.
Especially in the colder areas of the world it'd be criminal to waste the heat by pumping it outside (up to a certain point, anyway)
You could be heating buildings or a greenhouse with it, after all. Or making steam to pipe heat. Maybe even turning generators? Not sure what the step-down of the efficiency of it all is.
Apparently A/C is only 1/3rd efficient... but as you're going to be losing that anyway, might just look at the output heat.
Just once? I have a Slip and Slide between the racks!
The costs for improving data centers to provide more or colder air is more than just building out more square feet of data center space.
Just because HP is sells a 42U rack doesn't mean you have to cram blades into all 42Us. It's cheaper to spread the heat load across a larger area than to figure out how to put 1500 CFM out of a floor tile so the top of a blade rack gets air.
There are studies by the uptime institute that say that 50% of hardware failures happen in the top 25% of rack space because the top of the rack doesn't get any air from the floor tiles and it cycles exhaust from the rack or ambient air for cooling.
We just put in the latest blade rack from HP. 4 50 amp circuits(2 for redundancy) for a 4 square foot space is beyond silly. That's more service and electrical consumption than a 1500 square foot home after you eliminate the two circuits for redundancy.
STFU & GBTW
"Hey! Did you know that when you slashdotted that server near the Ross Ice Shelf, you caused 2 icebergs to calve? You insensitve clod!!!!"
Don't blame Durga. I voted for Centauri.
OK, here's a concept.
If data center location isn't such a problem as long as we have high speed data lines, locate the data center someplace nice and cold.
Like Manitoba, or Minnesota, or Alaska, or Siberia. Heat the work area with the flow from the data center.
Hire your local population to maintain the data center.
Profit !
There really isn't a question of if it will become widespread. Overclocking sites have had more than a few visits from Intel and AMD over the years. It's an inevitable problem with an inevitable solution. The only question is how long until water cooling becomes more popular. Heat needs have had people clamoring for Pentium M processors for rack mount gear for a while as well. It's a reasonably speed CPU that handles heat fairly well. It would work very nicely in rack mount gear, but motherboards that will take one are fairly rare.
As for server rooms, they will continue to be air conditioned for as long as all of your server room equipment is in their. Even if you found a magical solution for servers you still have RAID arrays, switches, routers and the like all in the same room. Server rooms are well known by HVAC people as requiring cooling. Most HVAC vendors will prioritize a failed server room HVAC over anything but medical. They know damn well that anybody that has an air conditioner designed to work in the middle of January in Minnesota or North Dakota isn't using the cooling for comfort.
2. Open the windows
3. Profit!!!
My beliefs do not require that you agree with them.
About 4 years ago, I was touring the US National Supercomputing Center in San Diego. One of the supercomputers had a clear plexiglass side where you could see inside, and it had running water and even a waterfall. Mind you, this 'water' was running directly over the electronic components. So the guy doing the tour said that it wasn't really water, but a chemical compound similiar to water, but very nonconductive. He tells us that it costs $10,000 per barrel, and that he always gets questions about what happens if you drink it. "Well, we're not sure what happens if you drink it, but we figure one of two things will happen. It could be toxic, and you drink it and die. Or, it could be nontoxic, and when our finicial guys found out you were drinking their $10,000-a-barrell water, they'll kill you."
To make laws that man cannot, and will not obey, serves to bring all law into contempt.
--E.C. Stanton
Data centers with heat problems usually fall into three categories; those with inadequate cooling capacity, those with inadequate cooling distribution, and those with unrealistic equipment densities.
However, I often find people have misconceptions, they think they have a heat problem, but in reality they do not. One must measure the air temperature at the inlet to the servers, not the exhaust. If the inlet air meets the manufacturer's specifications, there is no problem, despite the fact that it's uncomfortably hot in the exhaust aisle.
"Hot spots" can often be corrected by rebalancing, which is the science of redirecting the supply air proportionately to the heat loads in the space. Any good maintenance firm that knows data centers will offer rebalancing services.
If you really do have a heat load problem, e.g. more load than capacity, as evidenced by excessive temperatures throughout the space, consult a mechanical engineer that specialzes in data centers.
This isn't the sig you're looking for... Move along.
When I worked at Target we had specialized monitoring equipment that notified the same people that handle burglar alarms if a server room got to be too hot. It was written directly in the contract we had with the HVAC co's that only 911 call centers and Hospital Emergency rooms could be prioritized over one of our server rooms.
In a rack of 1U units, does each 1U slab take 240volt (or 115 or whatever) each, and have its own PSU?
I've often though it might be nicer if there could be one power supply for a whole room of PCs for example. This could be placed somewhere outside of the room and cooled there, with appropriate UPS backup too.
12 and 5 volt lines then feed each PC - no noisy or hot PSU per machine... Peace and quiet, and a bit cooler too...
No Norm, those are your safety glasses; I'll wear my own thanks...
We're currently going through re-evauating our cooling needs in our server room. The answer we came up with is we have to buy a bigger a/c unit.
Unfortunately, a couple times per year, the chilled water to our a/c unit gets shut off, and our servers are left to fry. The better answer is to have machines which run cooler. If they lose a/c, they won't fry. However, replacing clusters isn't cheap...and I don't think most people think, "which one's going to run the coolest?" when they are going to buy one.
Does anyone have a link to a page that has grossly generalized heat numbers on certain processor families in certain case configurations(I realize these numbers aren't going to be anywhere near exact, but it would be a starting point)?
PHB: Dear god, that server is actually red hot!
SA: Yes, but notice that the room is lovely and cool.
PHB: That's all right then. By the way, what's delaying that upgrade to Windows 2003?
SA: Every time we put the CD in the drive it melts. We think it's going to be fixed in the next service pack.
~~~~~ BigLig2? You mean there's another one of me?
I came in a few weeks ago and found my office full of water from where the water main into the boiler had cracked and formed a small lake in my office.
"**laughs** oh, I know, we'll stick the IT guy in the lake"
you have any suggestions on how to better protect equipment in those type situations?
Telecom equipment runs off -48VDC, and the phone company uses big batteries as their UPS.
It exists, it just is expensive.
When SGI added BX2 nodes to NASA's Columbia system, the standard air cooling was inadequate. They were forced to do a quick cooling change that added water cooling. Some would call the change a kludge.
More detail on the change, and cooling in general, can be found in this interview with the SGI designers who dealt with the problem.
Reduce power consumption
Reduce heat in the server room
Improve reliability
Best regards.
Bob was changing backup tapes when something caught his eye at his feet. Looking through te holes in the cooling tile in the raised floor, something was moving, like a bundle of shiny snakes. Looking closer, we had 1/2" of water down there!
We spent several hours with a tiny shop vac (we need a bigger one!) emptying the water and being thankful Bob had seen it before it got high enough to get into the power conduits.
An A/C unit drain pan had a clogged drain, so the sump pump couldn't carry the water away. Whoever had the units installed had purchased water alarms, but *they had never been hooked up*. Now *that* was a brilliant move.
We now have water alarms down there.
Meanwhile, the room stays about 70 degrees, and the servers stay comfy, as do we. I like it that way,
...aaaaaand where do you think that energy goes?
[DING] "Heat, Alex" "Correct, for $100."
...aaaaaand what do you think that energy loss thanks to high current means?
[DING] "Efficiency less than a modern AC->DC power supply" "Correct, for $200."
Anyone particpating in the "DC versus AC" discussion would do well to pick up a history book and read about Westinghouse and Edison. There's a reason we use A/C everywhere except for very short hauls. Modern switching power supplies are very efficient and still the best choice for this sort of stuff.
Please help metamoderate.
The industry has taken a two-pronged approach. Equipment vendors have been developing cabinets with built-in cooling, while design consultants try to reconfigure raised-floor data center space to circulate air more efficiently. The problem usually isn't cooling the air, but directing the cooled air through the cabinet properly.
There was an excellent discussion of this problem last year at Data Center World in Las Vegas. As enterprises finally start to consolidate servers and adopt blade serves (which were overhyped for years), many are finding their data centers simply aren't designed to handle the "hot spots" created by cabinets chock full of blades. Facilities with lower ceilings are particularly difficult to reconfigure. The additional cooling demand usually means higher raised floors, which leaves less space to properly recirculate the air above the cabinets. Some data center engineers refer to this as "irreversibility" - data center design problems that can't be corrected in the physical space available. This was less of an issue a few years back, when there was tons of decent quality data center space available for a song from bankrupt telcos and colo companies. But companies who built their own data centers just before blades became the rage are finding this a problem.
RichM
Data Center Knowledge
Because rarely is the AC ever plugged into the UPS (takes too much power) and most server rooms die during a power failure not due to the UPS running out of power, but because the room overheats and the servers all shut down.
Server rooms can turn into tropical saunas pretty fast. During a power failure we have to get into the office in 40 minutes to start powering down less important servers (try telling management that *all* the servers aren't mission critical, or worse yet, getting them to fork out $$$$$ for a bigger UPS)
Of the many server rooms I've been in, the most effective cooling I've seen has been to enclose the racks into sealed cabinets (adding a cheapish layer of physical security as well, by locking the things) and then piping cooled air directly into the top of the cabinets.
:)
If you buy your own racks to put gear in, then getting these things is easy, if you buy whole racks from a vendor with gear in it already (custom systems type of thing), then the thing comes in a cabinet which usually has some kind of a fan/vent arrangement on top. Rip that off, attach some ducting straight up to the ducts running across the rows, and voila, cool air flows straight down and out the bottom.
All you need is to build your room with several ducts running across the ceiling, with removable plates every so often. The AC system pushes air into that, which then goes directly into the racks. You don't even need to cool the room really, since the air coming out of the racks gets cool enough to keep the room itself cool. The servers in the racks stay at fairly chilly temp in there. Only downside is when you need to open one, you're hit in the face with this freezing air pouring out of the rack.
- Give a man a fire and he's warm for a day, but set him on fire and he's warm for the rest of his life.
We need to do everything to reduce the power required for all our electronic gear. More powerful servers [computational wise] require more power [electricity wise] which then requires more power [electricity wise] to power the air conditioning. If we could get server that somehow consumed less power [a lot less] we would win on two fronts.
Most internet data centers are not equipped with the cooling to handle customers with racks full of blades. A rack of HP BL40p blades puts out 55000 BTUs. A tier 1 data center in which I've worked was designed to cool 5000 BTUs per rack. While blades are pushed as a way to save space by increasing computing density, the amount of cooling per square foot of data center space, unless it has specifically been designed for blades, is rarely sufficient for cooling them. The aforementioned data center has more power available than it can cool. When sales gets its way (as it almost always does), more power is delivered to customer per square foot than the data center was designed for, then the room warms up, and everyone bitches.
It is right and proper for a data center to make it difficult to allow a customer to go that far out of the specs that the data center can support, or it'll negatively impact the other customers in the room. If they can't meet your needs, it's better to look elsewhere than to go that far out of engineering spec.
500GB of disk, 5TB of transfer, $5.95/mo
Use fresh, already cool outside air? In HVAC terms, you are referring to what's called an "economizer". These are not uncommon (and will save money for your typical building. Adding the necessary ducting, filtering and control hardware will add to the intitial cost, however.
For a datacenter, economizer options are not popular. Why? As a general rule, outside air is generally considered too dirty and too humid to be worth using on valuable data center equipment.
What would Groucho do?
The only effective way to cool 52 KW racks
is with a cooler in front of the rack -
This unit requires a 15 ton capacity coil,
but can be lined up side by side. Each rack stays cool.
Smaller units work the same way -For example- put a 4 ton coil in front of a rack of 40 opterons - cold air in the front regardless of the room temperature. 80 opterons / 8 ton coil The computers provide the fan - no crac unit needed -
Size the cooler for the rack, hook it to the cold water, - repeat as needed.
This adds 8 to 14 inches to rack depth - and has no hot spots.
email has changed - rcbondsr@gmail.com
For license information - University of Washington Technology Transfer.
Bigger computers - bigger coil - one rack at a time I can cool any installation you have !
I am going to roll out the first installation this summer.
From my patent application -
Background of the Invention
[003]Computers are sometimes cooled by cooling the air in the room in which the computers are located. A typical cooling system cools air and moves it through the room with the devices in the room that need to be cooled. When air is used as the cooling medium, variations in airflow occur, particularly when the heat density rises in a region of the equipment room, or when the absolute heat load approaches the maximum load that the air can handle. In an effort to solve resulting problems, systems have been made in which the devices that heat up are placed inside of a closure and the air inside the enclosure is cooled. These systems have been found to be inadequate when the heat density is above about 8 Kw. None of the existing systems are able to effectively operate in an environment in which the heat density is between about 20 to about 40 Kw. Yet, manufacturers are starting to make computer equipment in which that much power exists in the system. Currently, when the heat density is high, the systems are provided with greater floor space and larger air handlers and chillers. This approach has led to the creation of "hot spots" in the equipment. The known systems fail when the power level raises to about 400 watts per square foot, or when the cooling requirements vary substantially in a given space.
[004]When airflow in a single rack approaches about 3,000 cubic feet per minute, and an aisle of about 20 racks approaches 52,000 cubic feet per minute, the conventional systems cannot handle the airflow in a computer room of conventional size. The use of larger rooms is expensive and they are still subject to the airflow problems that are created. These problems include the creation of "hot spots" which are regions in the room that are not sufficiently cooled and in which the devices that generate the heat are adversely affected by the heat. There is a need for a cooling system that avoids the problems of the prior art systems and which eliminates the "hot spots". A principal object of the present invention is to fulfill this need.
ABSTRACT
A device that in use generates heat is positioned within an enclosed space that includes an ambient air inlet, an outlet and an air mover for moving ambient air through the space from the ambient air inlet to the outlet. A cooler comprising coils and passage ways defined by and between the coils to which ambient air moves from the inlet of the cooler to the outlet of the cooler. Position in the cooler with its outlet in register with the ambient air inlet for the enclosed space. Using the cooler to cool the ambient air that is immediately forwardly of the ambient air inlet for the enclosed space. Using an air mover in the enclosed space for moving the cooled ambient air into the ambient air inlet, through the enclosed space, and out from the outlet of the enclosed space.
rcb
We create our society every time we interact with each other. What kind of society did you create today?