Slashdot Mirror
Intel and SGI Test Full-Immersion Cooling For Servers
itwbennett (1594911) writes "Intel and SGI have built a proof-of-concept supercomputer that's kept cool using a fluid developed by 3M called Novec that is already used in fire suppression systems. The technology, which could replace fans and eliminate the need to use tons of municipal water to cool data centers, has the potential to slash data-center energy bills by more than 90 percent, said Michael Patterson, senior power and thermal architect at Intel. But there are several challenges, including the need to design new motherboards and servers."
The Cray-2 did this in 1985 using a liquid called Fluorinert also invented by 3M:
http://en.wikipedia.org/wiki/Cray-2
http://en.wikipedia.org/wiki/Fluorinert
A small history lesson for those who don't know, this is not the same SGI (or Silicon Graphics) than of the graphics workstation fame. This one is Rackable Systems which acquired the assets of the original SGI in 2009 (and SGI Japan in 2011).
Many years ago I invested in a Hardcore Computer Reactor system. This was a giant custom built computer that had both the motherboard and GPUs submerged in a proprietary non-conductive coolant. It weighs over a hundred pounds filled, and they still needed pumps inside it to direct the coolant across a bunch of purpose built water blocks to extract heat from the hottest components (since liquid convection alone was not enough).
About a year ago I had to replace the motherboard (which is a proprietary part). I can't even begin to tell you what a gigantic pain in the ass this was. There is a ton of plumbing running around inside the system that you have to worry about, and beyond that the entire compute module comes out of the coolant dripping wet, so you can't just pop it out and chuck it down on your desk. I had to break out a pair of rubberized gloves just to service the damned thing since it became obvious that the boards weren't going to dry themselves just sitting there- the coolant doesn't evaporate at all and you can't just take a towel to the raw PCB to clean it off. I landed up lining the inside of a large plastic bin with antistatic bags and doing the procedure there, which still made one hell of a mess.
I still run that system, but if anything else ever breaks I'm probably going to sell it off rather then try to fix it again. I honestly can't imagine trying to deal with that sort of a setup on a datacenter scale. General liquid cooling is easy enough to deal with since you can just disconnect the cooling lines and pull out a module (which is precisely what IBM does with their extreme high-end end PowerPC based servers). Submerging the entire PCB is nasty business, and I wouldn't want to be the tech who has to go through that amount of trouble on a weekly or monthly basis.
(sorry for the duplicated posting; the previous one was cut because of problems with the html marks)
In order to obtain a 90% reduction in the energy bill, cooling must account for 90% of the power of the DC. This implies a PUE >= 10. As a reference, 5 years ago virtually any DC had a PUE lower than 3. Nowadays, PUE lower than 1.15 can be obtained easily. As a referecence, Facebook publishes the instantaneous PUE of one of its DC in Prineville, which at the moment is 1.05. This implies that any savings in cooling would reduce the bill, at much, in a factor of 1.05 (1/1.05 = 0.9523).
On the other hand, I believe that this is not the first commertial offer for a liquid-cooled server, Intel was already considering two years ago, and the idea has been discussed in other forums for several years. I can't remember right now which company that was actually selling these solutions, but I believe it was already in the market.
None of the articles I've seen mentioned which version of Novec is being used. They have a great variety: http://solutions.3m.com/wps/po...
Sure, mineral oil, cooking oil, fluorinert distilled water, bunch of other esoteric fluids. The real thing that it comes down to the heat transfer between the component and the fluid itself. And this newer stuff is apparently leaps above flurorinert, especially besides that it won't kill you quite so quickly and won't destroy the ozone layer quite so badly. You thought that freon was bad? Fluorinert makes freon look like a glass of water in terms of reactivity.
Om, nomnomnom...
Yep. Got to fiddle around with Fluorinert cooling years ago.
Interesting, just not very practical.
You really DO need a fully sealed system and ostensibly clean-room assembly. Because, while the coolant itself is non-conductive, any detritus that accumulates in the fluid after settling out of the environment ISN'T. That's the main thing about water (straight H2O) isn't conductive. It's all the other things in the water, minerals, dust, etc that's doing the conduction.
Also, as noted, there's STILL going to be use of fans and water. Because you still need systems that extract the thermal energy from the liquid medium. You simply remove them from the main system chassis.
It also doesn't change the fact that it's still a TERRIBLY inefficient way to cool the system. Unlike water cooling loops, where you have no more than maybe a pint or so of fluid cooling the major heat sources in the system, you have QUARTS of fluid basically covering everything. And you really have no good flow control, other than extremely high volume fluid exchange, which is energy inefficient in and of itself.
That's PROBABLY what a lot of the board re-engineering is about. Centralizing all the thermally active devices into a centralized area to limit the volume of immersion coolant required and to simplify flow control.
Chas - The one, the only.
THANK GOD!!!
<quote><p>Sure, mineral oil, cooking oil, fluorinert distilled water, bunch of other esoteric fluids. The real thing that it comes down to the heat transfer between the component and the fluid itself. And this newer stuff is apparently leaps above flurorinert, especially besides that it won't kill you quite so quickly and won't destroy the ozone layer quite so badly. You thought that freon was bad? Fluorinert makes freon look like a glass of water in terms of reactivity.</p></quote>
Eh, no. While it isn't the nicest fluid available it is pretty much inert under normal circumstances,which strangely is the reason why the name ends with -inert.
Yes, it can. I've got a little bitcoin miner chip running right now as a proof of this. I'm not a bitcoin enthusiast, just wanted something hot and expendable to test immersion cooling on.
There is one downside: Viscosity. It's thick stuff, so it takes a powerful pump to keep it actively circulating. It also tends to pool in spaces underneath components and anywhere not exposed to easy circulation, impeding cooling.
it can be an advantage, as long as it doesn't break down on boiling.
that way the cpu can stay at 49c and the system can be built to not require pumps, just by piping the steam to a cooling tower and from tower back to servers. however of course this needs redesign of the server and components, like said.
world was created 5 seconds before this post as it is.
Even though according to wonkypedia it has low GW potential and doesn't damage ozone, do we really want to be manufacturing more fluorinated hydrocarbons which almost never decay in the enviroment by themselves and just build up over time in the soil, plants and eventually us?
There's still a minimum conductivity in molar water. But it's several orders of magnitude lower than than tap or bottled water.
Again, primary conductivity of water is via impurities in the water, not the water itself.
Chas - The one, the only.
THANK GOD!!!
Their fluid is boiling, phase transition takes a lot of heat out without pumping anything.
If not, you need to pump fluid between boards, this require more space and energy, even more with a thicker fluid.
49C is rather low for a boiling point for this application.
Is it? Phase transitions generally require quite a lot of energy. It is my understanding that if you allow the vapors to condense externally and return the liquid back, you'll get a significantly improved heat transfer. In fact, this is why heat pipes work so well.
Ezekiel 23:20
Air cooling is inefficient, but it's not so horrible that that inefficiency alone accounts for 90% of data center power usage. Heat is heat, and Watts is Watts; they gotta go somewhere.
And the "tons of water" that data centers use is generally used to spray the outdoor condenser (think cooling tower at a power plant); changing the servers to liquid cooling won't fix that.
Liquid cooling makes less sense for smaller servers, as going to all the trouble to plumb a pizza box is generally more trouble than it's worth. Big Iron is already frequently liquid cooled, if not in an immersion bath.