Slashdot Mirror
Startup's Submerged Servers Could Cut Cooling Costs
1sockchuck writes "Are data center operators ready to abandon hot and cold aisles and submerge their servers? An Austin startup says its liquid cooling enclosure can cool high-density server installations for a fraction of the cost of air cooling in traditional data centers. Submersion cooling using mineral oil isn't new, dating back to the use of Fluorinert in the Cray 2. The new startup, Green Revolution Cooling, says its first installation will be at the Texas Advanced Computing Center (also home to the Ranger supercomputer). The company launched at SC09 along with a competing liquid cooling play, the Iceotope cooling bags."
Until you have to try and RMA that CPU :)
the new Xeon 5600's run at less power than previous CPU's. and SSD's also run a lot cooler. how much does this liquid cooling enclosure cost and what is the performance compared to just upgrading your hardware?
HP is going to ship their Xeon 5600 servers starting on the 29th
..computers, allow me to label this a "fad"
The idea is funky, but to get good cooling you want convection (every joule of pump energy from a circulating pump gets transferred into the oil at yet more heat) which means deep tanks which means, to the server environment, goodbye high density.
The ONLY thing that has changed since I was doing this is the affordability of SSDs, which mean that now it is practical to immerse the whole computer, and the mass storage too, which makes things a lot simpler and cheaper, and means you really can be JUST oil cooled, not oil cooled mainly, except for air cooled HDs etc.
TOP TIP from an old hand.
If you are going to oil cool by immersion, buy the latest top quality hardware, because once immersed it stays there, you'll only pull it once to see why it sucks.
BIGGEST mistake experimenters make is using old hardware, cos you always end up playing with it, making mess, ahh fsckit..
Nota Bene if you are building one of these in anger, make allowances for the significant increase in the weight that the oil makes.
HTH etc
http://slashdot.org/~GuyFawkes/journal
How much harder does it make doing standard move cables/switch harddrives/change components maintenance?
One of the advantages of a standard rack to me is that all of that is fairly easy and simple, so you can fix things quickly when something goes wrong.
'Sensible' is a curse word.
In any kind of a large data center environment the whole floor is going to be covered in that shit in short order. I can just imagine the fun of dealing with workman's comp claims every other week because someone slipped on liquid coolant on the floor and injured themselves. Even with high quality components, if you have 30,000 servers in a big room, you're going to have someone out there fiddling with one or more of them on a daily basis, and keeping things clean when they're all fully immersed like that would be next to impossible, especially if you're dealing with oil.
There are other ways to make data center cooling more efficient, such as hot aisle containment and individual rack-top coolers blowing cold air directly in front of the racks. There's no reason a modern data center needs to move entire buildings full of air anymore, even without liquid cooling.
Oil immersion may or may not be more efficient, but it doesn't seem like it would scale well. In a large data center where some hardware component is failing on a daily basis, because you have tens of thousands of servers, keeping all that oil contained within the enclosures would be a major challenge. During maintenance, that stuff is going to be getting all over everything, including the tech, who can easily spread it all over anything he touches before he gets around to cleaning up. You'd need a cleaning crew out on the floor constantly.
Hard disks aren't sealed, there's always (at least, on the dozens of disks I've taken apart) a little felt-pad or sticker covered vent on them. I figured it was for equalisation or something crazy, but I'm not positive.
Given hard disks aren't sealed, wouldn't they fill with fluid and assuming they'd still function with a liquid screwing up the head mechanism (given modern disk's head's float above the platter surface on a cushion of air) wouldn't the increased viscosity slow down seek events?
Won't these servers bathed in oil still have the same thermal output? I don't understand why it would be cheaper to cool oil than it would air or any other medium..
I went to eat some animal crackers and the box said, "Do not eat if seal is broken." I opened the box and sure enough..
I'm also curious- is there any kind of fire hazard doing this on a large scale?
There isn't a lot to burn in a normal computer(at least not burn really well) but could a short circuit near a leak lead to a inferno in an oil cooled data centre?
Or is the oil treated in some way to make it less likely to burn?
With 30,000 servers in a big room, you do NOT want anyone "fiddling" with them at all. They need to be removed from the room and taken someplace else to be "fiddled" on.
Here's an idea. This would require a chassis redesign, but it would remove the maintenance problems mostly.
Make a special case for each system, which has no fans (since they're only useful for air-cooled systems), and has some type of pump for circulating the cooling oil. In this circulation loop is a heat exchanger, one built into each chassis. The backside of the chassis has two quick-connect connectors for connection to a cooling water supply. These are the type of connectors that close when they're unplugged. Such connectors are both on the water supply, and on the chassis. This way, when a server malfunctions, all the tech has to do is unplug it and pull it out of the rack. The water connectors will disengage, so only a few drops of water will spill (which will evaporate quickly). All the cooling oil will be contained within the server chassis.
The server can then be taken to a designated maintenance area where the oil can be drained and the server operated on, and then refilled with oil and plugged back into the server rack.
A server with this Intel Atom equipped mobo draws something like 25-35W under full load. And the performance of these D510 dual core processors is comparable to better Pentium 4 processors.
You don't need oil-air heat exchangers, oil vats, or anything of the kind. What you need is chilled WATER, which is already generated by cooling plants. Run this water to each server using simple pipes and a large pump for the whole facility, and then put an oil/water heat exchanger inside each chassis, along with a pump to circulate the oil.
Is the efficiency going to be better? Maybe, maybe not, who cares. What's different is that cooling is much easier with 3/8" pipes of water rather than worrying about ductwork and A/C units. This will also allow you to have much, much higher server density than with air cooling; fluid is a much better (and denser) conductor of heat than air. Instead of wasting a lot of space on fans and ductwork and other places for air to flow, you only have to worry about some little pipes. Floor space is expensive in a facility like this.
And if you keep the cooling oil contained within the servers, you won't have to worry about any mess.
Cue tech in scuba gear swimming down through the oil to change a power supply.
Interested in a Flash-based MAME front end? Visit mame.danzbb.com
I seem to remember mainframes using distilled water for cooling decades ago. Not being a member of the correct priesthood, I was not allowed in the mainframe room, so I don't know how it was set up then. I have seen how oil-filled systems work, and I would hate to work on one. Nasty mess.
Why, without your clothes, you're naked, Miss Dudley!
Educate thyself: http://en.wikipedia.org/wiki/Mineral_oil#Mechanical.2C_electrical_and_industrial
Just because something CAN burn doesn't make it dangerous to have around potential sources of electrical arcing. Hydrocarbon petroleum products present no real fire/explosion danger unless the substance is warmer than its flash point, which is the temperature above which the liquid substance can evaporate into the air. Below the flash point temperature, oil is only as flammable as plastic. The evaporated fumes mixed into the air are the ignition danger, not the liquid itself.
This is because ongoing hydrocarbon combustion requires steady supplies of freely-mixing HC and oxygen. Sustaining the reaction requires the input of a tremendous volume of oxygen (compared the the liquid fuel volume, anyway), and the oxygen has to get rapidly mixed with the HC. That mixing can't happen quickly enough to the liquid HC. That's why the flash point is such an important consideration--the gaseous HC fumes mix quite well and quickly with atmospheric oxygen, creating nice conditions for a sustained combustion (a fire).
This is even true of gasoline (flash point = -40F). If you pour gasoline into a pail in the middle of a bad Antarctic winter, and you throw a match into the pail, the gasoline will just extinguish the match like a bucket of water.
Of course, if you mix liquid HC with liquid oxygen, or any other eager oxidizers, all bets are off. That shit will explode at cryogenic temperatures if you just look at it funny. (That's how rocket engines work.)
The Cray 2 had a three stage cooling system; the flourinert was pumped through a heat exchanger and dumped it's heat into chilled water, which was either provided by the site's existing HVAC infrastructure or (more likely, since the dissipation was in the Megawatt range) by a dedicated freon-based water chiller. The 5th generation Cray Inc (as opposed to CCC) also used immersion cooling in a similar vein. Many other Cray machines (YMP, C90 and so on used the same 3-stage cooling system, but the modules weren't immersed in the flourinert, rather the coolant flowed through channels in a thermally conductive plate sandwiched between the two boards of each processor or memory module. This wasn't a means of cooling the boards more cheaply; this was ECL logic... in those days it was the only way you could deliver the required power and have the thing not literally melt.
Most power transformers are oil cooled. In every substation there are a few big ones, and there are many smaller ones on pole tops or on the ground in suburbs. They pump the oil through the transformer and into a radiator that may or may not be fan cooled. If you build it right, sometimes you dont even need a pump, you can just use the changing density of oil as it heats to have it move itself through the loop. Cooling computers would use the same principle. Oil is a good insulator. There is a certain amount of fire hazard, especially since an arc through the oil will break it down into gasses like acetylene and hydrogen. I'm sure on youtube there are some rather spectacular videos of transformer fires. However there are ways to mitigate the fire risk, and oil cooling is a rather old and well known technology. It has been used in the power industry for more than 50 years.