Slashdot Mirror

← Back to Stories (view on slashdot.org)

Supercomputing Cluster Immersed In Oil Yields Extreme Efficiency

Posted by samzenpus on from the staying-cool dept.

1sockchuck writes: A new supercomputing cluster immersed in tanks of dielectric fluid has posted extreme efficiency ratings. The Vienna Scientific Cluster 3 combines several efficiency techniques to create a system that is stingy in its use of power, cooling and water. VSC3 recorded a PUE (Power Usage Efficiency) of 1.02, putting it in the realm of data centers run by Google and Facebook. The system avoids the use of chillers and air handlers, and doesn't require any water to cool the fluid in the cooling tanks. Limiting use of water is a growing priority for data center operators, as cooling towers can use large volumes of water resources. The VSC3 system packs 600 teraflops of computing power into 1,000 square feet of floor space.

67 comments

  1. Immersing them in oil is fine... by 93+Escort+Wagon · · Score: 2

    ... but rubbing them with cheetah blood makes them even faster.

    --
    #DeleteChrome
    1. Re:Immersing them in oil is fine... by Anonymous Coward · · Score: 0

      Ah, I've heard of that traditional Chinese technique. It works just as well as the rhino horn.

  2. teraflops in oil by turkeydance · · Score: 1

    a fetish?

    1. Re:teraflops in oil by Anomalyst · · Score: 1

      a fetish?

      I put on my robe and wizard's hat ...

      --
      There is no right to feel safe thru security vaudeville at the expense of everyone's freedom, privacy and tax money.
  3. Dedicated, highly trained staff by TWX · · Score: 3, Informative

    This is fine when the application is high-end and the organization can justify employing expensive, highly-trained staff to maintain the equipment, but for the average business, even one with a fairly extensive datacenter, there is no practical way to do this. Too many staff do double-duty (ie, take care of software applications and occasionally do hardware maintenance) or are too reliant on software/systems vendors that are themselves locked in to hardware incompatible with such an environment for this to be practical. It's cool, but it's definitely a niche within a niche.

    --
    Do not look into laser with remaining eye.
    1. Re: Dedicated, highly trained staff by itomato · · Score: 2

      That's right.

      The company you'll see in the 2020 Fortune 250 who deals with all that nastiness is being formed right now because you get the willies.

    2. Re:Dedicated, highly trained staff by ClickOnThis · · Score: 1

      Agreed.

      Still, you have to admit that it brings new meaning to the phrase: "Do you want fries with that?"

      --
      If it weren't for deadlines, nothing would be late.
    3. Re:Dedicated, highly trained staff by JimSadler · · Score: 1

      If quality equipment is purchased and oil cooling is optimal i doubt that internal maintenance will be much of an issue at all. Oil immersed gear if free of dust and runs cool therefore components tend to be very long lasting. No fans or moving parts may be needed at all. I have experience with cooling towers for industrial chillers and the amount of water used can easily exceed the cost of electricity to cool a large building. I am aware of one school that had worn out baffles in an AC cooling tower and used $80,000 in city water in one month. using water from a lake or stream is usually forbidden by environmental codes as the heat released by exhaust water tends to change the entire ecology of the water source.

    4. Re:Dedicated, highly trained staff by Anonymous Coward · · Score: 0

      Bullshit. We've dunked our compute gear in a warehouse full of tanks of corn oil for years. It's exactly like it looks, horizontal racks, dust lids on hinges, etc. We've got better custom crossflow plumbing. And NO water evap units. It's ambient, ground loop, cold loop, and heat pump for water and hvac. 100degF or less, nonstop compute. You don't need special staff to play in oil, just an apron, some grubby clothes and the washer we provide if you mess yourself.

    5. Re:Dedicated, highly trained staff by cheater512 · · Score: 1

      Then a hard drive fails and someone has to get very wet and make a big mess to replace it.

    6. Re:Dedicated, highly trained staff by Trongy · · Score: 1

      Keep the hard drives immersed in air and connected via the SAN

    7. Re:Dedicated, highly trained staff by Rhys · · Score: 1

      Its cute but you're going to get failures. I hope they did some really amazing burn-in tests before immersing. We had 5% defective on arrival ram (ECC issues -- cuts a nodes gflops to 2/3 to 1/2 original) from a certain fruit vendor, using their gear in a supercomputer I ran. Most of the lifetime issues were just drives dieing, but we regularly also lost CPUs, myrinet boards, fan units, and even the occasional motherboard (the NICs tended to die). There was always something new dead every week or two while we ran that thing for ~5 years (too long in the HPC world; should have been replaced at 3-4 years).

      I can't imagine how much (more) not-fun that would have been had the whole thing been in an oil bath.

      --
      Slashdot Patriotism: We Support our Dupes!
  4. I use whale oil by Anonymous Coward · · Score: 1

    I'm all for saving the environment :)

    1. Re:I use whale oil by Anonymous Coward · · Score: 0

      FUckuh you WhALE!!

  5. Would not the oil start dissolving the parts? by coffecup · · Score: 1

    Would not oil start dissolving all the parts as it would be harsh on glues and plastics that most components are made up of?

    1. Re:Would not the oil start dissolving the parts? by joaommp · · Score: 3, Informative

      It's an inert type of oil developed by 3M for exactly this purpose.

      --
      Onda Technology Institute
    2. Re:Would not the oil start dissolving the parts? by ShanghaiBill · · Score: 2

      It's an inert type of oil developed by 3M for exactly this purpose.

      In addition to being chemically inert, a good liquid coolant should have other properties:

      1. Electrically insulating (duh).
      2. Low viscosity, so it flows easily.
      3. High thermal conductivity.
      4. High thermal expansion, to increase natural convection.
      5. Low vapor pressure.
      6. Non-flammable.
      7. Non-toxic.
      8. Cheap.
      Liquid fluorocarbons have often been used. But the fluid in this case is not a fluorocarbon. It is specially blended white mineral oil.

    3. Re:Would not the oil start dissolving the parts? by rmdingler · · Score: 1
      We are quite adept at job-specific innovation, especially that of insulation from the solvent-like characteristics of the oil(s) used as liquid refrigerant.

      Breaking the Computer Room Air Conditioning habit is a cute meme, but the transfer of heat through liquid refrigerant versus air is much more efficient.

      Think of the difference in your after party, immersing your hand in 232 Celsius versus 100 Celsius water.

      --
      Happiness in intelligent people is the rarest thing I know.

      Ernest Hemingway

    4. Re: Would not the oil start dissolving the parts? by Anonymous Coward · · Score: 0

      A better comparison might be air to water.

    5. Re:Would not the oil start dissolving the parts? by rmdingler · · Score: 1
      Damn.

      I just Trumped that up. I meant to say oven air after 232 Celsius.

      --
      Happiness in intelligent people is the rarest thing I know.

      Ernest Hemingway

    6. Re:Would not the oil start dissolving the parts? by Anonymous Coward · · Score: 0

      Pretty sure it would be an inert oil. Instead feel sorry for the guys that need to wade through 100m of oil to change a part.

    7. Re:Would not the oil start dissolving the parts? by guruevi · · Score: 1

      Non-flammable, non-toxic and cheap aren't really necessary for these things. Nobody is going to light a flame, drink it and the cost is usually offset by the savings.

      --
      Custom electronics and digital signage for your business: www.evcircuits.com
    8. Re:Would not the oil start dissolving the parts? by catsRus · · Score: 1

      Electrical Line transformers have been oil cooled for a long time, also dummy loads for RF testing. Lots of prior art for cooling electrics and no damage to packaging etc.

    9. Re:Would not the oil start dissolving the parts? by ClickOnThis · · Score: 1

      feel sorry for the guys that need to wade through 100m of oil to change a part.

      From the pictures in TFA, it appears the oil and the servers are contained in tanks that are about the size and shape of a small dumpster. Swapping out a server would entail removing it vertically via its handles. So, it would seem the ickiness is kept to a minimum. Certainly no wading through 100 m of oil.

      --
      If it weren't for deadlines, nothing would be late.
    10. Re:Would not the oil start dissolving the parts? by colin_faber · · Score: 1

      Sort of, not really an oil. It's a drivitive of Fluorinert which was developed by 3M and used extensively on Cray systems. The problem remains, once you have component failure you impart carcinogens into the fluid making it highly toxic. The other problem with this stuff is the price. Originally priced around $2K a litre (systems usually used hundreds of gallons), and now I've heard pricing down to $150/litre and some times cheaper. In any case it's neat looking stuff when you get to watch it boil away http://www.unixgr.com/wp-conte...

    11. Re:Would not the oil start dissolving the parts? by Anonymous Coward · · Score: 0

      No its not. Its a different fluid called electrosafe.

      the 3M stuff is intended for 2 phase cooling. This electrosafe stuff do not evaporate at all its a liquid all the time.

    12. Re:Would not the oil start dissolving the parts? by Anonymous Coward · · Score: 0

      Non-flammable, non-toxic and cheap aren't really necessary for these things. Nobody is going to light a flame, drink it and the cost is usually offset by the savings.

      As a safety precaution, non-toxic and non-flammable are wise concerns especially given Bill above's 1-8 enumeration lending to lower priority. They are wise, wisdom is a necessity in life. Wise is necessary.

      As for cheap... Free would be best. Cost is a valid concern since there is no benefit to spending more than necessary. Unless you are an Apple product consumer, those people think the more you pay the cooler it is.

      distrowatch.com Linux is free, and it's best. Supercomputers are ALL on Linux, none on Apple. None on Windows either. top500.org

      If there is to be a discovery of what is "best", there should be plans along the way to make it a "safe best" in various unforeseen environments. You wouldn't want a new oil-cooled PC set up later in some shop that has some accident and it's Chernobyl. [exaggerated, yes] Of course nobody is going to eat the PC.

      Why would anybody even counter that list of 1-8. It makes sense just fine.

    13. Re:Would not the oil start dissolving the parts? by chrism238 · · Score: 1
      > Would not oil start dissolving all the parts

      Have you seen the depreciation rate on supercomputers? If it's a race between the market price and the oil, my money's on the market.

    14. Re:Would not the oil start dissolving the parts? by kylemonger · · Score: 1

      You want the stuff non-flammable because you'll be sinking electrical equipment in the goo, so sparks need to not make the whole computer room go boom.

      You want the stuff non-toxic because in the event of an inadvertent leak or a disgruntled employee with an axe, you don't create an instant Superfund site. Nor do you want the people who maintain the racks to need to wear hazmat suits.

    15. Re:Would not the oil start dissolving the parts? by delt0r · · Score: 1

      Your not going to be allowed to use it in volume outside a tank farm if its flammable. Fire is something you don't intend to happen but does anyway.

      --
      If information wants to be free, why does my internet connection cost so much?
    16. Re:Would not the oil start dissolving the parts? by Anonymous Coward · · Score: 0

      You're thinking of the low boiling point Novec variants with boiling point in the 50-70C range at atmospheric pressure. Those are indeed aimed at 2-phase cooling.
      The heavier "stuff" with boiling point north of 140C? Very much intended for liquid cooling.

    17. Re:Would not the oil start dissolving the parts? by guruevi · · Score: 1

      Sparks in oil? You could soak the thing in diesel fuel or even gasoline, as long as it is fully submerged and contained there is not enough oxygen to ignite it and sparks do not develop in an oil based substance (again, you need oxygen to get a flame).

      I agree it shouldn't be hazardous but "toxic" does not always mean it's a hazard. There are berries that are toxic to humans yet totally fine in nature. Also, there are other containment methods and cleansing procedures similar to how underground gas station tanks operate.

      --
      Custom electronics and digital signage for your business: www.evcircuits.com
    18. Re:Would not the oil start dissolving the parts? by guruevi · · Score: 1

      As long as the oil remains contained within the unit, there is not a problem. Most oils aren't immediately flammable either. By the time most oils have the time and temperature to catch a flame, the sprinkler systems should have activated.

      --
      Custom electronics and digital signage for your business: www.evcircuits.com
    19. Re:Would not the oil start dissolving the parts? by ChrisMaple · · Score: 1

      9. Low dielectric constant. No need to slow down signals.

      --
      Contribute to civilization: ari.aynrand.org/donate
    20. Re:Would not the oil start dissolving the parts? by Anonymous Coward · · Score: 0

      Do you want fires with that?

    21. Re:Would not the oil start dissolving the parts? by delt0r · · Score: 1

      Sprinkler system on flamable oil. You really have no idea about fire regulations do you.

      --
      If information wants to be free, why does my internet connection cost so much?
    22. Re:Would not the oil start dissolving the parts? by guruevi · · Score: 1

      Water sprinklers in datacenters? You really have no idea about data centers do you?

      --
      Custom electronics and digital signage for your business: www.evcircuits.com
    23. Re:Would not the oil start dissolving the parts? by delt0r · · Score: 1

      Which is why its flammability will totally affect the permits on your ability to use the said oil dumbass. if it is not flammable and non toxic, such permits and handling procedures are easy and cheap. Which is why you want it to be non flammable! Back to the original post. Sheesh talking to wall here.

      --
      If information wants to be free, why does my internet connection cost so much?
    24. Re:Would not the oil start dissolving the parts? by guruevi · · Score: 1

      You seem to have failed basic physics and chemistry.

      As I said:
      Oil is not flammable, it's combustible, big difference.
      No oxygen means no flame, regardless of the fuel
      Datacenters remove the oxygen in case of fire, the fire suppression system uses an inert gas to displace oxygen.

      You do not need permits to handle oils in these situations. You are not 'storing' oil.

      --
      Custom electronics and digital signage for your business: www.evcircuits.com
    25. Re:Would not the oil start dissolving the parts? by delt0r · · Score: 1

      Flammable Combustible? Really? I think your brain fell out. Good luck explaining that after the fact when they find you didn't comply with the legal requirements.

      --
      If information wants to be free, why does my internet connection cost so much?
  6. Re:nike air max 90 france pas cher Femme by Anonymous Coward · · Score: 0

    /. rel=nofollows the links so, your spam is pretty pointless...

    Also, /. is full of nerds, nerds don't exercise, or need a basketball shoe.

  7. kept cool with water but does not user water by Anonymous Coward · · Score: 0

    "Take a supercomputer and immerse it in tanks of liquid coolant, which must be kept cool with the use of water."
    "It is particularly impressive given that it uses zero water."

    ??

    1. Re:kept cool with water but does not user water by penguinoid · · Score: 1

      Presumably it uses a closed loop cooler, rather than evaporative cooling. Evaporative cooling is very energy efficient, but sacrifices water for those energy savings.

      Presumably another benefit is that due to the improved heat transfer from components to the cooling oil, not only are the components cooler but you can run the cooling system at a much higher temperature and skip the fans besides.

      --
      Don't waste your vote! Vote for whoever you want, unless you live in a swing state it won't matter anyways
    2. Re:kept cool with water but does not user water by Anonymous Coward · · Score: 1

      Presumably it uses a closed loop cooler, rather than evaporative cooling. Evaporative cooling is very energy efficient, but sacrifices water for those energy savings.

      Presumably another benefit is that due to the improved heat transfer from components to the cooling oil, not only are the components cooler but you can run the cooling system at a much higher temperature and skip the fans besides.

      Use of water is not a big deal everywhere. Nice in deserts and such, but here I can look at all the river water that just flows by. And all the unwanted rain. You can get (untreated) river water for free. You don't water your lawn, you drain it to prevent it turning into a swamp. And still the shops market low-flush toilets and water saving diswashers that are only useful in other countries. Some idiots thinks it is a "greener way", failing to see how that only applies to places with actual water shortages.

      No, keep using cheap water. When the population is food-limited instead of water-limited, water saving is senseless.

  8. New? Really? by pz · · Score: 1

    The Cray-2 was a much earlier large-scale use of computational elements immersed in inert cooling liquid.

    --

    Put my fist through my alarm clock with its ding-dong death inside my ear. - The Blackjacks.
    1. Re: New? Really? by Anonymous Coward · · Score: 0

      IBM did it first. "Stretch" computer used oil cooled magnetic core memory back in early 60s. 96kbytes of memory in a box about 4ft x 6ft x 8ft with a big radiator on the back end to cool off the oil.

  9. Finally a solution to the California drought by Anonymous Coward · · Score: 0

    Instead of all the server farms sucking up water, they can use cheap abundant oil made from people.

  10. Re:New? Really? by thinkwaitfast · · Score: 1
    The problem with that was if the power went out, the Cray would over heat.

    Note, this was pointed out to the professor in an Electrical Properties of Materials class by a classmate.

  11. But does it? by Anonymous Coward · · Score: 0

    Also make delicious fries?

  12. Liquid Coolants by Taco+Cowboy · · Score: 1

    There have been plenty of 'submerge cooling' stories online, but most of the time the 'liquid coolants' turned out to be some exotic chemicals (flourine or something) which are very expensive and some are actually very bad for the environment

    Has anyone here run any kind of 'comparison test' on those liquid, on the level of their efficiency, cost, and whether or not they are environmentally safe or not?

    --
    Muchas Gracias, Señor Edward Snowden !
    1. Re:Liquid Coolants by ArhcAngel · · Score: 2

      I have been looking at 3M's NOVEC line of products for a couple of years. 3M claims their fluids are safe and I've seen it demonstrated a couple of times. I'm wondering how this compares. I've seen it for sale for around $250 gallon so it aint cheap but the cooling savings over time should more than pay for itself.

      --
      "A person is smart. People are dumb, panicky dangerous animals and you know it." - K
  13. Safety Hazard... by HockeyPuck · · Score: 3, Informative

    I worked with a company that did a proof of concept for these "oil bath systems" It was just like in the photo, a rack on its' back immersed in a tank of oil.

    From a cooling standpoint, it was great, from a parts replacement, it's a disaster waiting to happen. It's not how hard it is to replace the part. We installed a small crane above the rack to make it easier to lift the server up out of the bath, so we could lay it down and replace mem/cards/SSD etc. A 1RU server can weigh anywhere between 30-40lbs, and a 2RU server can weigh almost 70lbs.

    We had to surround the bath will a perforated rubber mat, as oil+typical datacenter floor is a huge slip hazard.

    You've also got to install pumps and a heat-exchanger. Reminds me of the old water cooled mainframes. Just with way more plumbing.

    1. Re:Safety Hazard... by Anonymous Coward · · Score: 0

      This is not for a typical datacenter. This is for HPC. These folks are used to dealing with exotic gear.

    2. Re:Safety Hazard... by Rhys · · Score: 1

      HPC gear isn't that exotic -- not since about 2000. Sure, some may still use crazy custom interconnect and motherboard solutions, but most are just a Nu rackmount with infiniband and maybe a gpgpu or two.

      --
      Slashdot Patriotism: We Support our Dupes!
    3. Re:Safety Hazard... by benthurston27 · · Score: 1

      The stuff they are probably using for this is the 3m engineered fluids that have a really high vapor pressure so the mess should just evaporate, and it's really low viscosity so the heat moves away from the parts by convection so no need for pumps, the heat exchanger is put above the fluid where it recondenses and falls back into the bath...

  14. Re:New? Really? by behrooz0az · · Score: 2

    Wouldn't the Cray turn off at the same time?

    --
    Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion. -- Spazmania (174582)
  15. Re:New? Really? by Anonymous Coward · · Score: 0

    Erm, but if the power went out, the Cray would be offline too ?
    If not, then the same emergency power the Cray is kept running on could be used to keep the cooling system online..

  16. Y not F by Anonymous Coward · · Score: 0

    This story makes so much more sense now that I realize I misread it being in oil fields.

  17. Everything old is new again by Required+Snark · · Score: 1
    IBM 7302 Core Memory Unit circa 1957-1958 for the IBM 7030 (Stretch) computer.

    The core memory in the IBM 7302 was heated/cooled to stabilize its operating characteristics. Early units immersed the core stack in heated/cooled oil, later units called the IBM 7302A, blew heated/cooled air through the core stack.

    I once herd a story from a real old timer about fixing these memories. When they were new there was a problem with small metal particles left over from manufacturing floating around and shorting out the core stacks. In those days IBM field engineers always wore white shirts and a tie. When they had to fix these things, they would take off their tie, but just leave their shirts on and pull the core out with their hands, because they knew that they would get covered in oil no matter what they did.

    --
    Why is Snark Required?
    1. Re:Everything old is new again by DarthVain · · Score: 1

      Heck, not even that long ago OC folks have been testing with the stuff for years. I recall reading several articles years ago, where people were taking souped up (pardon pun) PC's and immersing them in a liquid solution to test for cooling... They may or may not be using more less the same inert liquid.

      As I recall, while there were some benefits, there were significant problems as well. Maintenance was obliviously one, both of the components, but also the bath. Also it should be pointed out, while this does allow all components to be cooled at the same time, certain components heat up significantly faster than others, so unless you have significant circulation (and coolant), all you are doing is heating up the surrounding components.

      Presumably this is an exotic enough system that the expense of maintaining it isn't an issue.

  18. Have I missed something by JohnStock · · Score: 1

    Didn't Cray and others do this decades ago?

  19. link to its homepage by Anonymous Coward · · Score: 0

    the vienna supercomputing effort:

    http://vsc.ac.at/

    its last cluster - vsc3

    http://vsc.ac.at/systems/vsc-3/

    cheers
    v.

  20. Conductivity by Anonymous Coward · · Score: 0

    "Servers are inserted vertically into slots in the tank, which is filled with 250 gallons of ElectroSafe fluid, which transfers heat almost as well as water but doesn’t conduct an electric charge."

    Water does not conduct electric charge either.

    1. Re:Conductivity by ledow · · Score: 2

      Maybe pure water, but handling pure water is a pain in the butt and the slightest contamination will make it conductive.

  21. I, for one... by Anonymous Coward · · Score: 0

    I, for one, welcome our new oil-immersed overl... hey, wait...