Slashdot Mirror


Supercomputing Cluster Immersed In Oil Yields Extreme Efficiency

1sockchuck writes: A new supercomputing cluster immersed in tanks of dielectric fluid has posted extreme efficiency ratings. The Vienna Scientific Cluster 3 combines several efficiency techniques to create a system that is stingy in its use of power, cooling and water. VSC3 recorded a PUE (Power Usage Efficiency) of 1.02, putting it in the realm of data centers run by Google and Facebook. The system avoids the use of chillers and air handlers, and doesn't require any water to cool the fluid in the cooling tanks. Limiting use of water is a growing priority for data center operators, as cooling towers can use large volumes of water resources. The VSC3 system packs 600 teraflops of computing power into 1,000 square feet of floor space.

45 of 67 comments (clear)

  1. Immersing them in oil is fine... by 93+Escort+Wagon · · Score: 2

    ... but rubbing them with cheetah blood makes them even faster.

    --
    #DeleteChrome
  2. teraflops in oil by turkeydance · · Score: 1

    a fetish?

    1. Re:teraflops in oil by Anomalyst · · Score: 1

      a fetish?

      I put on my robe and wizard's hat ...

      --
      There is no right to feel safe thru security vaudeville at the expense of everyone's freedom, privacy and tax money.
  3. Dedicated, highly trained staff by TWX · · Score: 3, Informative

    This is fine when the application is high-end and the organization can justify employing expensive, highly-trained staff to maintain the equipment, but for the average business, even one with a fairly extensive datacenter, there is no practical way to do this. Too many staff do double-duty (ie, take care of software applications and occasionally do hardware maintenance) or are too reliant on software/systems vendors that are themselves locked in to hardware incompatible with such an environment for this to be practical. It's cool, but it's definitely a niche within a niche.

    --
    Do not look into laser with remaining eye.
    1. Re: Dedicated, highly trained staff by itomato · · Score: 2

      That's right.

      The company you'll see in the 2020 Fortune 250 who deals with all that nastiness is being formed right now because you get the willies.

    2. Re:Dedicated, highly trained staff by ClickOnThis · · Score: 1

      Agreed.

      Still, you have to admit that it brings new meaning to the phrase: "Do you want fries with that?"

      --
      If it weren't for deadlines, nothing would be late.
    3. Re:Dedicated, highly trained staff by JimSadler · · Score: 1

      If quality equipment is purchased and oil cooling is optimal i doubt that internal maintenance will be much of an issue at all. Oil immersed gear if free of dust and runs cool therefore components tend to be very long lasting. No fans or moving parts may be needed at all. I have experience with cooling towers for industrial chillers and the amount of water used can easily exceed the cost of electricity to cool a large building. I am aware of one school that had worn out baffles in an AC cooling tower and used $80,000 in city water in one month. using water from a lake or stream is usually forbidden by environmental codes as the heat released by exhaust water tends to change the entire ecology of the water source.

    4. Re:Dedicated, highly trained staff by cheater512 · · Score: 1

      Then a hard drive fails and someone has to get very wet and make a big mess to replace it.

    5. Re:Dedicated, highly trained staff by Trongy · · Score: 1

      Keep the hard drives immersed in air and connected via the SAN

    6. Re:Dedicated, highly trained staff by Rhys · · Score: 1

      Its cute but you're going to get failures. I hope they did some really amazing burn-in tests before immersing. We had 5% defective on arrival ram (ECC issues -- cuts a nodes gflops to 2/3 to 1/2 original) from a certain fruit vendor, using their gear in a supercomputer I ran. Most of the lifetime issues were just drives dieing, but we regularly also lost CPUs, myrinet boards, fan units, and even the occasional motherboard (the NICs tended to die). There was always something new dead every week or two while we ran that thing for ~5 years (too long in the HPC world; should have been replaced at 3-4 years).

      I can't imagine how much (more) not-fun that would have been had the whole thing been in an oil bath.

      --
      Slashdot Patriotism: We Support our Dupes!
  4. I use whale oil by Anonymous Coward · · Score: 1

    I'm all for saving the environment :)

  5. Would not the oil start dissolving the parts? by coffecup · · Score: 1

    Would not oil start dissolving all the parts as it would be harsh on glues and plastics that most components are made up of?

    1. Re:Would not the oil start dissolving the parts? by joaommp · · Score: 3, Informative

      It's an inert type of oil developed by 3M for exactly this purpose.

    2. Re:Would not the oil start dissolving the parts? by ShanghaiBill · · Score: 2

      It's an inert type of oil developed by 3M for exactly this purpose.

      In addition to being chemically inert, a good liquid coolant should have other properties:

      1. Electrically insulating (duh).
      2. Low viscosity, so it flows easily.
      3. High thermal conductivity.
      4. High thermal expansion, to increase natural convection.
      5. Low vapor pressure.
      6. Non-flammable.
      7. Non-toxic.
      8. Cheap.
      Liquid fluorocarbons have often been used. But the fluid in this case is not a fluorocarbon. It is specially blended white mineral oil.

    3. Re:Would not the oil start dissolving the parts? by rmdingler · · Score: 1
      We are quite adept at job-specific innovation, especially that of insulation from the solvent-like characteristics of the oil(s) used as liquid refrigerant.

      Breaking the Computer Room Air Conditioning habit is a cute meme, but the transfer of heat through liquid refrigerant versus air is much more efficient.

      Think of the difference in your after party, immersing your hand in 232 Celsius versus 100 Celsius water.

      --
      Happiness in intelligent people is the rarest thing I know.

      Ernest Hemingway

    4. Re:Would not the oil start dissolving the parts? by rmdingler · · Score: 1
      Damn.

      I just Trumped that up. I meant to say oven air after 232 Celsius.

      --
      Happiness in intelligent people is the rarest thing I know.

      Ernest Hemingway

    5. Re:Would not the oil start dissolving the parts? by guruevi · · Score: 1

      Non-flammable, non-toxic and cheap aren't really necessary for these things. Nobody is going to light a flame, drink it and the cost is usually offset by the savings.

      --
      Custom electronics and digital signage for your business: www.evcircuits.com
    6. Re:Would not the oil start dissolving the parts? by catsRus · · Score: 1

      Electrical Line transformers have been oil cooled for a long time, also dummy loads for RF testing. Lots of prior art for cooling electrics and no damage to packaging etc.

    7. Re:Would not the oil start dissolving the parts? by ClickOnThis · · Score: 1

      feel sorry for the guys that need to wade through 100m of oil to change a part.

      From the pictures in TFA, it appears the oil and the servers are contained in tanks that are about the size and shape of a small dumpster. Swapping out a server would entail removing it vertically via its handles. So, it would seem the ickiness is kept to a minimum. Certainly no wading through 100 m of oil.

      --
      If it weren't for deadlines, nothing would be late.
    8. Re:Would not the oil start dissolving the parts? by colin_faber · · Score: 1

      Sort of, not really an oil. It's a drivitive of Fluorinert which was developed by 3M and used extensively on Cray systems. The problem remains, once you have component failure you impart carcinogens into the fluid making it highly toxic. The other problem with this stuff is the price. Originally priced around $2K a litre (systems usually used hundreds of gallons), and now I've heard pricing down to $150/litre and some times cheaper. In any case it's neat looking stuff when you get to watch it boil away http://www.unixgr.com/wp-conte...

    9. Re:Would not the oil start dissolving the parts? by chrism238 · · Score: 1
      > Would not oil start dissolving all the parts

      Have you seen the depreciation rate on supercomputers? If it's a race between the market price and the oil, my money's on the market.

    10. Re:Would not the oil start dissolving the parts? by kylemonger · · Score: 1

      You want the stuff non-flammable because you'll be sinking electrical equipment in the goo, so sparks need to not make the whole computer room go boom.

      You want the stuff non-toxic because in the event of an inadvertent leak or a disgruntled employee with an axe, you don't create an instant Superfund site. Nor do you want the people who maintain the racks to need to wear hazmat suits.

    11. Re:Would not the oil start dissolving the parts? by delt0r · · Score: 1

      Your not going to be allowed to use it in volume outside a tank farm if its flammable. Fire is something you don't intend to happen but does anyway.

      --
      If information wants to be free, why does my internet connection cost so much?
    12. Re:Would not the oil start dissolving the parts? by guruevi · · Score: 1

      Sparks in oil? You could soak the thing in diesel fuel or even gasoline, as long as it is fully submerged and contained there is not enough oxygen to ignite it and sparks do not develop in an oil based substance (again, you need oxygen to get a flame).

      I agree it shouldn't be hazardous but "toxic" does not always mean it's a hazard. There are berries that are toxic to humans yet totally fine in nature. Also, there are other containment methods and cleansing procedures similar to how underground gas station tanks operate.

      --
      Custom electronics and digital signage for your business: www.evcircuits.com
    13. Re:Would not the oil start dissolving the parts? by guruevi · · Score: 1

      As long as the oil remains contained within the unit, there is not a problem. Most oils aren't immediately flammable either. By the time most oils have the time and temperature to catch a flame, the sprinkler systems should have activated.

      --
      Custom electronics and digital signage for your business: www.evcircuits.com
    14. Re:Would not the oil start dissolving the parts? by ChrisMaple · · Score: 1

      9. Low dielectric constant. No need to slow down signals.

      --
      Contribute to civilization: ari.aynrand.org/donate
    15. Re:Would not the oil start dissolving the parts? by delt0r · · Score: 1

      Sprinkler system on flamable oil. You really have no idea about fire regulations do you.

      --
      If information wants to be free, why does my internet connection cost so much?
    16. Re:Would not the oil start dissolving the parts? by guruevi · · Score: 1

      Water sprinklers in datacenters? You really have no idea about data centers do you?

      --
      Custom electronics and digital signage for your business: www.evcircuits.com
    17. Re:Would not the oil start dissolving the parts? by delt0r · · Score: 1

      Which is why its flammability will totally affect the permits on your ability to use the said oil dumbass. if it is not flammable and non toxic, such permits and handling procedures are easy and cheap. Which is why you want it to be non flammable! Back to the original post. Sheesh talking to wall here.

      --
      If information wants to be free, why does my internet connection cost so much?
    18. Re:Would not the oil start dissolving the parts? by guruevi · · Score: 1

      You seem to have failed basic physics and chemistry.

      As I said:
      Oil is not flammable, it's combustible, big difference.
      No oxygen means no flame, regardless of the fuel
      Datacenters remove the oxygen in case of fire, the fire suppression system uses an inert gas to displace oxygen.

      You do not need permits to handle oils in these situations. You are not 'storing' oil.

      --
      Custom electronics and digital signage for your business: www.evcircuits.com
    19. Re:Would not the oil start dissolving the parts? by delt0r · · Score: 1

      Flammable Combustible? Really? I think your brain fell out. Good luck explaining that after the fact when they find you didn't comply with the legal requirements.

      --
      If information wants to be free, why does my internet connection cost so much?
  6. New? Really? by pz · · Score: 1

    The Cray-2 was a much earlier large-scale use of computational elements immersed in inert cooling liquid.

    --

    Put my fist through my alarm clock with its ding-dong death inside my ear. - The Blackjacks.
  7. Re:New? Really? by thinkwaitfast · · Score: 1
    The problem with that was if the power went out, the Cray would over heat.

    Note, this was pointed out to the professor in an Electrical Properties of Materials class by a classmate.

  8. Liquid Coolants by Taco+Cowboy · · Score: 1

    There have been plenty of 'submerge cooling' stories online, but most of the time the 'liquid coolants' turned out to be some exotic chemicals (flourine or something) which are very expensive and some are actually very bad for the environment

    Has anyone here run any kind of 'comparison test' on those liquid, on the level of their efficiency, cost, and whether or not they are environmentally safe or not?

    --
    Muchas Gracias, Señor Edward Snowden !
    1. Re:Liquid Coolants by ArhcAngel · · Score: 2

      I have been looking at 3M's NOVEC line of products for a couple of years. 3M claims their fluids are safe and I've seen it demonstrated a couple of times. I'm wondering how this compares. I've seen it for sale for around $250 gallon so it aint cheap but the cooling savings over time should more than pay for itself.

      --
      "A person is smart. People are dumb, panicky dangerous animals and you know it." - K
  9. Safety Hazard... by HockeyPuck · · Score: 3, Informative

    I worked with a company that did a proof of concept for these "oil bath systems" It was just like in the photo, a rack on its' back immersed in a tank of oil.

    From a cooling standpoint, it was great, from a parts replacement, it's a disaster waiting to happen. It's not how hard it is to replace the part. We installed a small crane above the rack to make it easier to lift the server up out of the bath, so we could lay it down and replace mem/cards/SSD etc. A 1RU server can weigh anywhere between 30-40lbs, and a 2RU server can weigh almost 70lbs.

    We had to surround the bath will a perforated rubber mat, as oil+typical datacenter floor is a huge slip hazard.

    You've also got to install pumps and a heat-exchanger. Reminds me of the old water cooled mainframes. Just with way more plumbing.

    1. Re:Safety Hazard... by Rhys · · Score: 1

      HPC gear isn't that exotic -- not since about 2000. Sure, some may still use crazy custom interconnect and motherboard solutions, but most are just a Nu rackmount with infiniband and maybe a gpgpu or two.

      --
      Slashdot Patriotism: We Support our Dupes!
    2. Re:Safety Hazard... by benthurston27 · · Score: 1

      The stuff they are probably using for this is the 3m engineered fluids that have a really high vapor pressure so the mess should just evaporate, and it's really low viscosity so the heat moves away from the parts by convection so no need for pumps, the heat exchanger is put above the fluid where it recondenses and falls back into the bath...

  10. Re:kept cool with water but does not user water by penguinoid · · Score: 1

    Presumably it uses a closed loop cooler, rather than evaporative cooling. Evaporative cooling is very energy efficient, but sacrifices water for those energy savings.

    Presumably another benefit is that due to the improved heat transfer from components to the cooling oil, not only are the components cooler but you can run the cooling system at a much higher temperature and skip the fans besides.

    --
    Don't waste your vote! Vote for whoever you want, unless you live in a swing state it won't matter anyways
  11. Re:New? Really? by behrooz0az · · Score: 2

    Wouldn't the Cray turn off at the same time?

    --
    Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion. -- Spazmania (174582)
  12. Everything old is new again by Required+Snark · · Score: 1
    IBM 7302 Core Memory Unit circa 1957-1958 for the IBM 7030 (Stretch) computer.

    The core memory in the IBM 7302 was heated/cooled to stabilize its operating characteristics. Early units immersed the core stack in heated/cooled oil, later units called the IBM 7302A, blew heated/cooled air through the core stack.

    I once herd a story from a real old timer about fixing these memories. When they were new there was a problem with small metal particles left over from manufacturing floating around and shorting out the core stacks. In those days IBM field engineers always wore white shirts and a tie. When they had to fix these things, they would take off their tie, but just leave their shirts on and pull the core out with their hands, because they knew that they would get covered in oil no matter what they did.

    --
    Why is Snark Required?
    1. Re:Everything old is new again by DarthVain · · Score: 1

      Heck, not even that long ago OC folks have been testing with the stuff for years. I recall reading several articles years ago, where people were taking souped up (pardon pun) PC's and immersing them in a liquid solution to test for cooling... They may or may not be using more less the same inert liquid.

      As I recall, while there were some benefits, there were significant problems as well. Maintenance was obliviously one, both of the components, but also the bath. Also it should be pointed out, while this does allow all components to be cooled at the same time, certain components heat up significantly faster than others, so unless you have significant circulation (and coolant), all you are doing is heating up the surrounding components.

      Presumably this is an exotic enough system that the expense of maintaining it isn't an issue.

  13. Have I missed something by JohnStock · · Score: 1

    Didn't Cray and others do this decades ago?

  14. Re:kept cool with water but does not user water by Anonymous Coward · · Score: 1

    Presumably it uses a closed loop cooler, rather than evaporative cooling. Evaporative cooling is very energy efficient, but sacrifices water for those energy savings.

    Presumably another benefit is that due to the improved heat transfer from components to the cooling oil, not only are the components cooler but you can run the cooling system at a much higher temperature and skip the fans besides.

    Use of water is not a big deal everywhere. Nice in deserts and such, but here I can look at all the river water that just flows by. And all the unwanted rain. You can get (untreated) river water for free. You don't water your lawn, you drain it to prevent it turning into a swamp. And still the shops market low-flush toilets and water saving diswashers that are only useful in other countries. Some idiots thinks it is a "greener way", failing to see how that only applies to places with actual water shortages.

    No, keep using cheap water. When the population is food-limited instead of water-limited, water saving is senseless.

  15. Re:Conductivity by ledow · · Score: 2

    Maybe pure water, but handling pure water is a pain in the butt and the slightest contamination will make it conductive.