Slashdot Mirror


New Approach To Immersion Cooling Powers HPC In a High Rise

miller60 writes "How do you cool a high-density server installation inside a high rise in Hong Kong? You dunk the servers, immersing them in fluid to create an extremely efficient HPC environment in a hot, humid location. Hong Kong's Allied Control developed its immersion cooling solution using a technique called open bath immersion (OBI), which uses 3M's Novec fluid. OBI is an example of passive two-phase cooling, which uses a boiling liquid to remove heat from a surface and then condenses the liquid for reuse, all without a pump. It's a slightly different approach to immersion cooling than the Green Revolution technique being tested by Intel and deployed at scale by energy companies. Other players in immersion cooling include Iceotope and Hardcore (now LiquidCool)."

63 comments

  1. Now Hiring: SCUBA certified network admins by Anonymous Coward · · Score: 2, Funny

    Waiting for a watered-down version for the consumer space.

    1. Re:Now Hiring: SCUBA certified network admins by noh8rz10 · · Score: 2

      didn't the computers and physchic people in minority report work like this? the future is now! Not to mention individualized advertising in stores, and gestures to control computers. I should rent this movie again... did it have a jetpack scene as well?

    2. Re:Now Hiring: SCUBA certified network admins by Anonymous Coward · · Score: 1

      Not to mention a mind control device that wrapped around the skull -> Google Glass....

    3. Re:Now Hiring: SCUBA certified network admins by Anonymous Coward · · Score: 0

      This idea was also found in the movie Sunshine. A truly terrible movie BTW, cool ideas but the execution of those idea was just... awful.

  2. Cray-3 used this by Anonymous Coward · · Score: 0

    http://en.wikipedia.org/wiki/Fluorinert
    Hope they don't have a fire.

    1. Re:Cray-3 used this by bobbied · · Score: 1

      Fluorinert wasn't flammable. At least the version they used on the C-90. (And I actually worked with these and other Cray hardware.. )

      --
      "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
    2. Re:Cray-3 used this by fuzzyfuzzyfungus · · Score: 2

      Fluorocarbons don't tend to be particularly flammable themselves; but the halogenated compounds not specifically designed for firefighting do sometimes have some...zesty...thermal decay products. They may or may not be self-sustaining reactions, and may even pull enough energy out of a would-be fire to help suppress it; but there are very few biological processes that are improved by adding improv halogen compounds...

    3. Re:Cray-3 used this by JayBat · · Score: 1

      Yup, if I'm the local fire department with one of these compute farms in my city, I really want to know what the decomposition products are...

    4. Re:Cray-3 used this by fuzzyfuzzyfungus · · Score: 2

      Since a wide variety of materials trade under 'Novec' with various model numbers, it's hard know which MSDS to link to; but hydrogen fluoride/hydrofluoric acid seem to be common to almost all of them, if thermally displeased, with some oddities like Perfluoroisobutylene and Perfluorinated acid fluorides for flavor, in certain compositions.

      All claim to be pretty well behaved at lower temperatures; but at 3-400 degrees (Celsius) I'm staying the hell away.

    5. Re:Cray-3 used this by N2717Y · · Score: 2

      Allied Control has been using Novec 649 which has the same MSDS as 3M Novec 1230 Fire Protection Fluid.

    6. Re:Cray-3 used this by mspohr · · Score: 1

      Bad for the firemen who have to go in and breath the toxins (or anyone else in the neighborhood of the fire).
      These chemicals should be banned.

      --
      I don't read your sig. Why are you reading mine?
    7. Re:Cray-3 used this by Anonymous Coward · · Score: 0

      Breathe? Hell, with HF around I don't want my skin exposed.

  3. Two phase is asking for trouble. by camperdave · · Score: 3, Informative

    Using a boiling liquid is asking for trouble. True, a phase change like boiling can conduct away a lot of heat, but there are other issues. First of all, vapour does not conduct as much heat as liquid, so there's going to be an insulating layer of vapour over all of the components that need it most. It's called the Leidenfrost Effect. Second, bubbles expanding and collapsing causes an effect called cavitation, which can erode components. It is a constant sonic vibration which can induce metal fatigue in delicate wires (such as the leads inside a chip), and can cause cracks in inflexible materials such as silicon chips and ceramics (capacitors and resistors).

    --
    When our name is on the back of your car, we're behind you all the way!
    1. Re:Two phase is asking for trouble. by sugarmatic · · Score: 5, Informative

      Nucleate boiling is what keeps the lights on if you depend on coal or gas for your electricity. It precedes the zone where your Leidenfrost effect is relevant, and actually increases the heat transfer coefficient by factors.

      Tuning a closed system to exploit this is an exercise (fluid chemistry, pressure, temperature), but it is also ubiquitous. As for cavitation, it's a red herring in the nucleate boiling zone- the size of the bubbles is so small, and hence the driving frequency is so high, there is a) less mechanical coupling for the vibration, and b) the energy of cavitation is so low as to not be an issue.

    2. Re:Two phase is asking for trouble. by UnknowingFool · · Score: 2

      I'm not sure that the Leidenfrost effect applies as much when all the components are immersed in liquid. It seems to be more of an issue when a hot surface is suddenly exposed to a small amount liquid. Second the corrosion effects you describe have more of an effect with water and sudden pressure changes. The Novec fluid used doesn't have the same properties as water.

      --
      Well, there's spam egg sausage and spam, that's not got much spam in it.
    3. Re:Two phase is asking for trouble. by DerekLyons · · Score: 4, Insightful

      The Liedenfrost Effect only really causes problems when you have to remove a great deal of heat very rapidly without pumps - I.E. inside a boiler. Inside a PC, it's very unlikely to be a problem. The same goes for cavitation, it's only a problem when there's a large number of bubbles forming and collapsing very rapidly or if there is a high pressure differential. Inside a PC, these conditions are unlikely to obtain and are thus unlikely to be a problem. (Not to mention, current PC's have constant vibration due to fans, hard drives, CD/DVD players, etc... and don't seem to suffer from any problems.)

    4. Re:Two phase is asking for trouble. by fuzzyfuzzyfungus · · Score: 2

      The naive overclock nuts who try liquid nitrogen on either a bare CPU die or a bare heatspreader sometimes discover the hard way that the Liedenfrost Effect is enough to generate a nice cushion of nitrogen between their couple-of-hundred-watts in a square centimeter or two CPU and the bulk of their liquid nitrogen; but that's mostly a surface area problem, which could just be solved with a basic heatsink, were they not trying to cut out every last component in the chain that has a nonzero delta-t.

    5. Re:Two phase is asking for trouble. by Anonymous Coward · · Score: 1

      Which is why heatpipes don't work at all and don't last for decades... oh, wait.

    6. Re:Two phase is asking for trouble. by K.+S.+Kyosuke · · Score: 1

      How do you explain the major success that is called the heat pipe, then?

      --
      Ezekiel 23:20
    7. Re:Two phase is asking for trouble. by CODiNE · · Score: 2

      Obligatory link to THE Leidenfrost effect paper.

      EPIC

      Do Not Try This At Home.

      But if you do...
      Try not to shatter your teeth.

      --
      Cwm, fjord-bank glyphs vext quiz
    8. Re:Two phase is asking for trouble. by DigiShaman · · Score: 1

      Most of the extreme (really extreme) OC nuts are not stupid. They just happen to have a LOT of money to burn. Quite literally. For these guys, it's like drag racking with top fuel cars. The engines are only good for a race or two before an entire teardown and rebuild is required. Interesting, but just as useless!

      --
      Life is not for the lazy.
    9. Re:Two phase is asking for trouble. by fuzzyfuzzyfungus · · Score: 1

      Oh, the non-naive ones don't make that mistake. It's the one with his first Dewar ever who has just finished gluing a length of pipe to his CPU socket that you have to watch out for.

    10. Re:Two phase is asking for trouble. by bluefoxlucid · · Score: 1

      That effect is called "X, but more slowly." For example: Nuclear explosion, bad. Nuclear explosion, but more slowly, awesome. Rapidly boiling ammonia, bad. Evaporating ammonia with a high partial pressure, good.

    11. Re:Two phase is asking for trouble. by camperdave · · Score: 1

      Easy. The fluid in a heat pipe is not in direct contact with the CPU. The metal of the heat transfer block and pipe is what is conducting the heat away from the CPU. Any Leidenfrost effect within the pipe merely insulates that section of pipe, and the heat simply conducts further down the pipe wall.

      --
      When our name is on the back of your car, we're behind you all the way!
    12. Re:Two phase is asking for trouble. by Anonymous Coward · · Score: 0

      And you're wrong again.
      Seriously, what kind of idiots mod this bullshit informative?

    13. Re:Two phase is asking for trouble. by camperdave · · Score: 1

      What part of what I wrote is wrong? Is the fluid in the heat pipe in direct contact with the CPU? Is the heat transfer block not metal? Does the heat not travel along the metal of the pipe wall until it reaches thermal equilibrium with the working fluid?

      --
      When our name is on the back of your car, we're behind you all the way!
  4. I've wondered why this isn't more common... by mlts · · Score: 1

    Most buildings have a core water chiller. It wouldn't be that hard to have a heat exchanger going to the fluid circulation system, then liquid cool every rack and item inside.

    The big problem is engineering the valves and connectors. It would be nice for a leak or an improper connection to be detected, and a valve shutting off coolant until it is fixed. Having quick-connect connectors which will shut off coolant flow when disconnected is also imperative. The goal would be for an almost immediate disconnection if someone cut a hose, ideally both active valves that shut off if there is a leak, as well as check valves so coolant cannot move backwards, so as little coolant as possible gets lost if something gets punctured.

  5. Controlling vapor loss? by fuzzyfuzzyfungus · · Score: 2

    TFA doesn't say exactly which 'Novec' heat transfer fluid they use; but all of them have enthusiastic vapor pressures even at 25 degrees, and low boiling points. The devices shown in the pictures (unless specially opened just for the glamor shots) also don't look particularly well sealed.

    How do they deal with that? Does the 3M guy follow the milkman every morning, and deliver another couple thousand liters? Is there a chiller/condenser somewhere in the air circulation system that scrubs most or all of the escaping vapor? Are the racks normally sealed tight?

    I, again, couldn't get any solid quotes for medium-large quantities of the heat transfer fluids; but fancy fluoridated-carbon engineered fluids aren't generally cheap enough to just ignore large losses of. Boiling may well be more efficient than pumping as a heat transfer method at the board level; but I'd be amazed if they can get away with running this as anything other than a closed loop, despite the pictures seemingly showing otherwise.

    Anybody know?

    1. Re:Controlling vapor loss? by Algae_94 · · Score: 1
      From the summary

      OBI is an example of passive two-phase cooling, which uses a boiling liquid to remove heat from a surface and then condenses the liquid for reuse, all without a pump

      It's light on details, but indicates that they are somehow reclaiming any boiled off fluid.

    2. Re:Controlling vapor loss? by ttucker · · Score: 1

      OBI is an example of passive two-phase cooling, which uses a boiling liquid to remove heat from a surface and then condenses the liquid for reuse, all without a pump.

      At least read the summary of the article.

    3. Re:Controlling vapor loss? by fuzzyfuzzyfungus · · Score: 1

      I read that claim in the summary; but was baffled by where and how the condense the liquid for reuse; because the pictures showed no signs of vapor capture or chiller/condenser units(at least nothing that looked similar enough to the ones I've seen for me to recognize them).

    4. Re:Controlling vapor loss? by N2717Y · · Score: 1

      Changing fluid or refilling is not necessary with well designed systems, because they are made to avoid or minimize losses (including vapor) or contamination. We found that spilling liquid while pouring in and out of containers is by far biggest source for losses, and we have integrated ways to avoid spills on initial filling, along with many other technical features that are designed to keep fluid in the tank.

      From Allied Contro's FAQs:
      http://www.allied-control.com/immersion-cooling/frequently-asked-questions-faqs/item/4-do-i-have-to-change-or-refill-the-liquid

      The key difference in open bath immersion cooling is to keep tanks under ambient pressure.

    5. Re:Controlling vapor loss? by Anonymous Coward · · Score: 0

      The vapor is condensed in the tank by facility water flowing through a condenser. The systems are sealed but maintained at atmospheric pressure. They actually consume very little fluid, like a vapor degreaser. This small technology demonstrator will show you how the tanks and control systems are built:

      http://www.youtube.com/watch?v=ZrPq4ltZa0g
      http://www.youtube.com/watch?v=ijs6sqTGLps

    6. Re:Controlling vapor loss? by N2717Y · · Score: 1

      Are you looking at the same pictures? At least half of the pictures is a condenser.

      http://www.datacenterknowledge.com/wp-content/uploads/2013/11/applied-control-novec.jpg

    7. Re:Controlling vapor loss? by fuzzyfuzzyfungus · · Score: 1

      I think those parallel green things are circuit boards boiling the coolant furiously. You can see the twisted wiring bundles running along the bottom edge of the picture, and what look like either interconnect headers or brass standoffs between the boards.

      The copper pipes are obviously involved in circulation in some way; but show absolutely no condensation, nor any fins or other surface-area-increasing features, so I'd be surprised if they are chillers.

      I think that that might be a shot of their FPGA array (since it's clearly a circuit board setup; but lacks any sign of standard PC or server features.)

    8. Re:Controlling vapor loss? by ttucker · · Score: 1

      Most of the pictures are of sealed tanks. The picture of fluid boiling is clearly labeled as being an image of what is inside of the sealed tanks. Certainly they have not shown every part of the system, but the photos shown certainly do not contradict the claims of the article.

    9. Re:Controlling vapor loss? by sugarmatic · · Score: 1

      It looks pretty darn well sealed to me. It's just a thermopile design. Phase change simply means a carefully designed system can get away without pumping a lot of fluid around.

      The stuff is expensive. A gallon or so can set you back several hundred dollars...and not new stuff either....that's a reclaimed cost.

  6. Slashdot = fookin' hoors? by Anonymous Coward · · Score: 1

    So, what, is Slashdot shilling for 3M now or something?
    Liquid cooling of hardware isn't a new idea at all. How is this even a news story?
    Come on /., how much did 3M pay you to whore yourselves out on this one?

  7. Get rid of fans, air spaces and gaps how small? by Virtucon · · Score: 1

    One thing that was mentioned was removing the air cooling parts from the servers. I'm wondering now if we'll see server vendors putting together configurations that will be specifically designed for immersion technology and if that will improve density / cubic meter in terms of rack space. Wouldn't this lead to higher density configurations, say 2 or 3 times current rack density or more? Also what about disk drives (rotating) can they be immersed?

    --
    Harrison's Postulate - "For every action there is an equal and opposite criticism"
    1. Re:Get rid of fans, air spaces and gaps how small? by sandytaru · · Score: 1

      Blade servers are already done in this stripped down format, with the PSU and cooling parts in the chassis. Problem is, they're prohibitively expensive right now. Since these immersed servers wouldn't require cases, I'd imagine they'd cost quite a bit less.

      --
      Occasionally living proof of the Ballmer peak.
    2. Re:Get rid of fans, air spaces and gaps how small? by fuzzyfuzzyfungus · · Score: 1

      Immersed HDDs aren't going to be happy. Barring some (very expensive) edge cases that are carefully sealed with a suitable gas mixture (so that the heads will continue to float above the platters at very high altitudes/low pressures and/or very high pressures won't cause increased drag, a niche that has probably been largely murdered by cheaper SSDs of late), HDDs aren't fully sealed. They have some aggressive dust filters, and the breathing hole isn't large, so infiltration might well be fairly slow; but once the fluid gets inside, it isn't going to be pretty.

      TFA mentioned some sort of FPGA cluster arrangement as being one of the early cases, which presumably didn't have a bunch of drives involved, except perhaps at one or more control head nodes, a more 'normal PC' centric arrangement would presumably either use PXE, iSCSI or fibre channel HBAs, or some other mechanism that allows the drive cage (which does need a bit of airflow; but nothing special) to be at a distance from the compute silicon, which is where the action is, thermally.

    3. Re:Get rid of fans, air spaces and gaps how small? by Anonymous Coward · · Score: 0

      You mean like the new 6Tb disks that are sealed and filled with helium?

      http://slashdot.org/story/13/11/04/1542230/6tb-helium-filled-hard-drives-take-flight

  8. Dunk your own computer! by bobbied · · Score: 2

    http://www.pugetsystems.com/submerged.php

    I always thought this was a cool idea, albeit pretty messy. But this company will sell you a "case" (a modified fish aquarium) and the components to go into it. They also have some specs on how well this thing cools.

    --
    "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
    1. Re:Dunk your own computer! by bmorency · · Score: 1

      Looking at the video the cpu and power supply fans are still attached and spinning. How does this affect the life of the fans as the are pushing around a liquid instead of just air? Do the fans do much as all the components are in the liquid? Does anyone know more about this?

    2. Re:Dunk your own computer! by fuzzyfuzzyfungus · · Score: 1

      The Puget arrangement uses a fluid with a much higher boiling point, so enthalpy of vaporization (and rapid changes in density) aren't carrying heat away as actively as in TFA's boiling coolant arrangement. Given that, I suspect that having fans/propellers pushing hot coolant out of the heatsinks is helpful. Since passive/convection cooling is possible with air, and fluid cooling is likely much more efficient, it'd presumably be possible with fluid as well; but 'possible' and 'easy' are not identical words, and fans are cheap insurance(especially if you don't have the space or interest to replace all existing heatsinks with huge passive ones, probably with more widely spaced fins than normal to handle a more viscous coolant fluid.

      As for lifespan? The bearings are probably kept pretty well lubricated (I'm sure that using mineral oil shocks and horrifies serious lubricant aficionados; but most fans make do with whatever they were given at the factory, and run until they dry out and die, so this probably counts as luxury for them), and the coolant fluid should help keep the electromagnets and driver circuitry from burning out. Brushless DC motors can deliver superb torque at low speed, so stalling shouldn't be an issue (though smarter motor control boards, if capable of sensing their own RPM, might freak out and shut down because they suspect that something is gravely wrong if they are spinning at least a factor of ten slower than designed).

      I have no reason to doubt that it works; but you'd have to be a real enthusiast to choose this approach over a simple liquid cooling loop on the CPU and GPU, with a big, slow, lazy, fan (200mm or two running at downright silent speeds) to keep random motherboard components and things from dropping dead inside an otherwise sealed case.

    3. Re:Dunk your own computer! by bobbied · · Score: 1

      Read their information page. The fans in the Oil do turn much slower than in air, but they are really not necessary when the whole thing is submerged in oil. They also have had a system in operation since 2007 with no fan failures, or any other failures. Apparently the oil dissipates any heating effects of turning slow. I wouldn't expect that the fans would ever wear out in this situation. When fans fail, they get noisy and then stop turning. The primary failure mode is the bearings wearing out due to dust build up in the bearings and the stress of imbalances due to dust on the blades. None of this will happen when dunked in oil.

      Somebody needs to figure out how to cool the oil below room temperature. If you could get things down to 0C, overclockers would have a hay day.

      --
      "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
  9. Some serious operational impacts by Anonymous Coward · · Score: 0

    If maintenance on a single unit with a rack is needed, wouldn't you have to shut down everything in the rack and drain the fluid bath? This approach would create the need for quite novel operational approaches.

    1. Re:Some serious operational impacts by PPH · · Score: 2

      Blade servers inserted vertically (like a server rack lying on its back)?

      --
      Have gnu, will travel.
    2. Re:Some serious operational impacts by JayBat · · Score: 1

      When a single unit in a rack fails, you just power it off. You don't bother draining the rack until some percentage of the units (5/10/15%, whatever) have failed, then you do them all at once (or more likely by that point, replace the entire rack). They're just interchangeable widgets...

    3. Re:Some serious operational impacts by fuzzyfuzzyfungus · · Score: 1

      For extra credit, spring-load the rails so that the server can 'pop' up into position for service (like a car trunk with pistons).

      Then tape a shark silhouette to the server, so it appears to be leaping out of the water at the unprepared service tech.

    4. Re:Some serious operational impacts by Anonymous Coward · · Score: 0

      Or just hot-swap components as it says on the Allied page. Novec 7000 boils or 34C and Novec 649 at 49C - a standard CPU is hotter than that...

  10. Coming Soon to a Movie Near You... by mythosaz · · Score: 2

    ...the bad guy will point his gun at the server room, threatening to shatter the glass case with the server cooling liquid in it, he'll shoot, and use the rushing liquid to block the heroes from stopping his escape.

  11. I tried this last year and it's kind of neat by Anonymous Coward · · Score: 0

    It's actually kind of neat to toss out your fans and just dunk your computer into a tank like it's a deep fryer. Of course, I didn't do this with my best build, instead I used older parts and made a pretty fun project. I no longer have any of those parts but it was a blast going against conventional logic. There are some parts that aren't meant to be dunked so keep that in mind if you value your life lol

  12. I know it is new but how does it relate to... by Anonymous Coward · · Score: 0

    what Cray used to do. And what IBM abandoned thirty years ago. Novec sounds like a very useful material for this sort of thing.

  13. But how does it work with storage media? by rosseloh · · Score: 1

    I mean, I imagine solid state storage would be fine, but I can't see a mechanical disk behaving too well in this sort of environment. Unless it's completely watertight, of course. And I don't see disks going away anytime soon.

    I suppose you could have a separate, air-cooled storage bay, but that would introduce new exciting engineering difficulties...


    (To be absolutely fair, I imagine something like this is going to cost a bit, and if you are spending that much already you might as well be springing for solid state storage...)

    1. Re:But how does it work with storage media? by fuzzyfuzzyfungus · · Score: 1

      Probably cheapest to go with a single, small, SSD onboard, that would then connect to networked storage for actual capacity.

      Going totally diskless, and doing an iSCSI or Fibre Channel HBA, would be more elegant; but you can probably get 128GB of decent quality SSD for less than a bootable HBA of either flavor would cost you, and that's easily enough space to install anything from a VMware hypervisor to a full Windows or Linux server capable of speaking almost any network storage protocol you want through cheap, commodity, ethernet.

    2. Re:But how does it work with storage media? by N2717Y · · Score: 1

      Not only solid state storage, also helium-filled hard drives (such as the WD 6TB Ultrastar He6).

    3. Re:But how does it work with storage media? by rosseloh · · Score: 1

      Thanks for that reference - I'd never heard of those drives before. Now I've got some learning to do.

  14. What's in a name by rokstar · · Score: 1

    Other players in immersion cooling include Iceotope and Hardcore (now LiquidCool).

    Who's principle leads are AcidBurn and CrashOverride respectively, no doubt.

  15. Where does the heat go? by Anonymous Coward · · Score: 0

    Located in a high rise, are they cooling and condensing the Novec fluid into the ambient air? If they're not dumping the heat outside, they'd be bad neighbors: externalizing the cost of cooling and cooking the people above them.

    1. Re:Where does the heat go? by Anonymous Coward · · Score: 0

      The top of each unit is a gas-liquid heat exchanger where the Novec condenses and the latent heat gets dumped into a water loop that runs to roof chillers.

  16. Cost vs Fluorinert by LoRdTAW · · Score: 1

    I wonder how Novec compares cost wise to Fluorinert. We use Fluorinert at work here and I know the larger FC-72 jugs are very costly at around $1500 and the FC-40 jugs are priced as if they were filled with liquid gold, over 3 grand and much smaller than the FC-72 jugs.