Slashdot Mirror


A Closer Look At Immersion Cooling For the Data Center

1sockchuck writes "Want to save money on data center cooling? Tip your racks on their side, fill them with mineral oil, and submerge your servers. Austin startup Green Revoluton Cooling first profiled here) has a video demo of its immersion cooling solution, which it says can handle racks using up to 100kW of power. A photo gallery on the company web site shows some early installations."

213 comments

  1. Mineral oil = nightmare by haxor45 · · Score: 0

    Didn't we discuss that before? One guy tried to use it on his rig (Link to his blog). It did work cool initially (no pun intended), but later it ruined his system as oil stated to leak. I bet they will run into same problems. (Imagine oil leaking out of Ethernet port...)

  2. Marketing that niche? by pieisgood · · Score: 1

    The thing here is they are commercializing a cooling technique usually reserved for the hobbyist. I don't know about the energy saving claims, but their setup looks fairly organized. Interesting turn for a still niche cooling solution.

    --
    Eat sleep die
    1. Re:Marketing that niche? by Anonymous Coward · · Score: 0

      Many supercomputers have been doing this for decades. It's hardly reserved for the hobbyist.

    2. Re:Marketing that niche? by Anonymous Coward · · Score: 0

      The thing here is they are commercializing a cooling technique usually reserved for the hobbyist. I don't know about the energy saving claims, but their setup looks fairly organized. Interesting turn for a still niche cooling solution.

      http://en.wikipedia.org/wiki/Cray_1

      Get off my lawn.

  3. Re:Mineral oil = nightmare by slim · · Score: 1

    Your link doesn't work, but I imagine it was a hobbyist. It looks as if this lot have built an industrial-strength product. It's clearly not practical in the home.

    The entire computer is submerged in a bath of oil, and the oil is circulated through a cooling tower. I doubt there are any holes for IO below the oil level of the bath, so leakage isn't a concern.

    I don't fancy the messy job of making hardware changes though.

  4. Re:Mineral oil = nightmare by Anonymous Coward · · Score: 0

    Error 404 - Not Found
    Sorry, the page that you are looking for does not exist.

  5. Liquid Cooling in Military Applications by Anonymous Coward · · Score: 0

    The thought of liquid cooling is nothing new. I know the us military uses water cooling aboard submarines for their server racks. In that app, however, they're pump water into those racks then use a heat exchangers to keep the equipment cool in an sealed rack. I'm a little shocked that using mineral oil really improves that power density THAT much for a massive data center. It just feels like cooling that much mineral oil can't be as efficient as cooling air.

    1. Re:Liquid Cooling in Military Applications by AlecC · · Score: 2

      I would expect oil to be far more efficient than air. It has a hugely greater thermal capacity (hundreds of times), so it can extract much more heat from the chips and similarly hand it over better to the cooling vanes. You use thermal paste to connect the chip to heat sinks better than air - this is a larger scale version of the same thing, where the whole system is immersed in a sort of thermal paste.

      --
      Consciousness is an illusion caused by an excess of self consciousness.
    2. Re:Liquid Cooling in Military Applications by Linuxmonger · · Score: 2

      You're right, it takes a lot of energy to heat up that much oil, that's the point. It doesn't take much energy to heat up a cubic foot of air, it takes a lot more to heat up a cubic foot of oil, the effort here is to remove as much of the energy that would go into melting chips as is possible.
      Liquid to air heat exchangers can be made as big as you need pretty cheaply, it's easier to pump 800 cubic feet of air through a big radiator cooling one cubic foot of oil than it is to pump 800 cubic feet of air through an OEM heat-sink.

    3. Re:Liquid Cooling in Military Applications by Anrego · · Score: 1

      From my experience building a hobby oil cooled pc .. I can imagine oil doing a much better job.

      Oil has a much greater heat capacity over air. What this translates to is the oil takes a _long_ time to heat up. In my 21 gallon rig, with no cooling or circulation... the oil just sitting still takes several hours to heat up to a dangerous temperature. Try turning off your PC fans and see how long the box lasts!

      The inverse is of course true as well.. once it heats up.. it stays hot for just as long.

      If you had an efficient way of keeping that oil cool and flowing at a decent rate... I can imagine this working quite well.

      For the curious, my oil rig worked great until the HSF popped off the processor one day (probably a combination of weird forces on the (very large) heat sink and maybe a cheat retention clip)... then it didn't. :(

  6. What about the hard drives? by Compaqt · · Score: 2

    Well, on the other hand, if they're supposed to be air-tight, I guess they're baby oil-tight, too.

    But there's got to be something or another that doesn't react well with mineral oil, right?

    I guess this means they save on fans, and the power to run fans. That's additional power and heat savings right there.

    OK, I've got it: what about the CD/DVD drives? Or is it all network IPL in data centers? I'm racking my brains trying to think of something this would mess up.

    --
    I'm not a lawyer, but I play one on the Internet. Blog
    1. Re:What about the hard drives? by slim · · Score: 1

      TFA says the hard drives have to be sprayed with a coating, presumably to make the housing oil-tight as well as airtight.

      Blowing air around is a tremendously inefficient way of cooling, and this replaces that with pumping the oil through a heat exchanger / cooling tower.

      Things with unsealed moving parts obviously are vulnerable, but not everything needs to be submerged. If you really want a DVD drive, have it outside the oil.

      There may be some things that don't react well with mineral oil: avoid having these in your system!

    2. Re:What about the hard drives? by gclef · · Score: 4, Informative

      Actually, hard drives are *not* supposed to be air tight. They intentionally allow airflow into the HD, but through a filter to keep dust out. If you want a drive that is airtight, it'll cost more.

      http://www.acsdata.com/how-a-hard-drive-works.htm

    3. Re:What about the hard drives? by petermgreen · · Score: 2

      TFA says the hard drives have to be sprayed with a coating, presumably to make the housing oil-tight as well as airtight.

      This should IMMEDIATELY ring alarm bells. Hard drives are NOT airtight. They have a filtered air hole. They would never get away with such flimsy construction on an airtight product.

      Plus does this system REALLY offer that much advantage over conventional "waterblocks" which keep the cooling fluid seperate from the electronics. I very much doubt it. The major heat generators in a PC are designed to pass out their heat by contact conduction anyway.

      --
      note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
    4. Re:What about the hard drives? by fuzzyfuzzyfungus · · Score: 2

      I've never tested to see if anything bad happens if you change this yourself; but hard drives are definitely not airtight. Pretty much all of them have one or more visible holes(usually on the top cover, often next to a "Do Not Cover" sticker). There is a fairly serious looking filter on the inside, covering the hole(some sort of fine carbonish powder sealed in what appears to be a teflon pouch); but that is to impede dust, rather than gases.

      Nothing stopping you from PXE booting(or iSCSI or fiber channel HBAs on the high end); but I'd be leery of running a hard drive under fluid, except as an experiment.

      The only other issue I can imagine might crop up would be discovering the hard way that some polymer used in one of the system's components doesn't handle oil exposure well in the long term. I suspect that most are fine; but if the plasticizer used to soften the insulator coating on some important bundle of wires leaches out over 18 months in a warm oil bath, and the embrittled insulator cracks and shorts the next time you mess with it, the joke would be on you...

    5. Re:What about the hard drives? by fuzzyfuzzyfungus · · Score: 3

      I suspect that the main advantage over waterblocks is that your server vendor of choice doesn't have to cooperate, and you don't have to run so many small hoses, and can get away with the big huge tank. Now, if some sort of critical mass were achieved in terms of industry acceptance of waterblocks as a factory option(along with some sort of standardized fluid connector), I suspect that that would swiftly become a superior option(though, quite possibly not displacing fans entirely. A fair number of ancillary systems, too small to justify their own waterblocks, get very unhappy if they aren't getting any airflow, even if the CPUs are taken care of...)

    6. Re:What about the hard drives? by admiralranga · · Score: 1

      A waterblock "plate" that sits flat over the mobo touching the heat generating componets, with groved water channels so the water goes in a loop (cpu first etc), with a water in and out that is connected to the rack, would do the job quite nicely i'd imagine.

    7. Re:What about the hard drives? by fuzzyfuzzyfungus · · Score: 1

      In systems not designed for it, that could get kind of ugly(DIMMs, riser cards, expansion cards parallel to the motherboard, etc) and having to re-CAD the part because your vendor went to rev A01 from rev A00 or they are now shipping memory modules with slightly different sizes would be a huge pain.

      I am told, though, that for very harsh environment embedded systems, that is pretty much what they do. Conformal heatsink plate firmly attached to the board so that the whole thing is a solid block.

    8. Re:What about the hard drives? by macs4all · · Score: 3, Interesting

      The only other issue I can imagine might crop up would be discovering the hard way that some polymer used in one of the system's components doesn't handle oil exposure well in the long term. I suspect that most are fine; but if the plasticizer used to soften the insulator coating on some important bundle of wires leaches out over 18 months in a warm oil bath, and the embrittled insulator cracks and shorts the next time you mess with it, the joke would be on you...

      Hard drives aren't the only thing designed with "vent holes".

      Every single electrolytic capacitor has a tiny vent hole (to keep them from acting like a mini fragmentation grenade if they develop an internal short circuit, etc.) Over time, with thermal cycling, the oil might get pumped in and out of the vent holes, thus degrading the electrolyte (guessing), and one fine day...

      And as you say, think of the insulation on the cables...

    9. Re:What about the hard drives? by Anonymous Coward · · Score: 1

      Perhaps it is time to invest in Solid State for booting and a SAN for storage?

    10. Re:What about the hard drives? by Smidge204 · · Score: 2

      Capacitors do not "breathe" like you describe, otherwise humidity would get in and ruin them. Instead they have "vents" in the sense that part of the casing is weakened to rupture safely if the electrolyte starts to break down and build pressure, rather than have the whole can explode.

      The irony here is that the electrolyte in the "beer can" style caps is a mineral oil not completely unlike what they use as a coolant. The other option is to pay extra for non electrolytic capacitors on your equipment.

      As for the oil degrading plastics and such, I'd like to think they thought of that and either use special wiring where it contacts oil or that the "special formulation" they speak of is designed to prevent that issue. Given that they warranty everything they're probably aware of something so obvious.
      =Smidge=

    11. Re:What about the hard drives? by camperdave · · Score: 1

      This, of course, doesn't apply to solid state hard drives.

      --
      When our name is on the back of your car, we're behind you all the way!
    12. Re:What about the hard drives? by drinkypoo · · Score: 1

      The irony here is that the electrolyte in the "beer can" style caps is a mineral oil not completely unlike what they use as a coolant. The other option is to pay extra for non electrolytic capacitors on your equipment.

      Is there a list of manufacturers using solid caps? My Gigabyte board proclaims loudly on the boot screen JAPANESE solid capacitor... whoops. Looks like it's time to find a new supplier. Having sucked up a lungful of blown cap smoke in the past, I'm glad to see solids...

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
    13. Re:What about the hard drives? by GameboyRMH · · Score: 1

      The other thing that rang my alarm bells is the idiocy of submerging hard drives at all. You DON'T DO IT. It doesn't help, it only causes problems. They should either do what immersion gaming cases do and use long SATA cables to run the hard drives to a dry compartment on the outside of the case, or network-boot these servers.

      --
      "When information is power, privacy is freedom" - Jah-Wren Ryel
    14. Re:What about the hard drives? by drinkypoo · · Score: 1

      I am told, though, that for very harsh environment embedded systems, that is pretty much what they do. Conformal heatsink plate firmly attached to the board so that the whole thing is a solid block.

      They used to do this on notebooks. I'm ripping apart a HP mobile P4 notebook for embedded use and I found out why it is so damned heavy, it has a single gigantic cast/milled heat sink which crosses CPU and GPU (old Radeon.) My later, bigger HP used heat pipes which actually made it lighter.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
    15. Re:What about the hard drives? by Anonymous Coward · · Score: 0

      Sure makes Solid State Drives more appealing.

    16. Re:What about the hard drives? by GameboyRMH · · Score: 1

      My understanding is that if you cover a hard drive's vent hole, unusual atmospheric pressures could cause the disk head to either crash into the platter or float too high over it.

      --
      "When information is power, privacy is freedom" - Jah-Wren Ryel
    17. Re:What about the hard drives? by Anonymous Coward · · Score: 0

      Most components get their heat out through the leads, except those designed for a heat sink (e.g. CPU, graphics processors, some FPGAs, power devices)
      Immersing in oil is a fine, but messy way to get higher density, at the cost of much lower maintainability.

    18. Re:What about the hard drives? by slim · · Score: 1

      ... or use SSD.

      But if you're going to have HDD storage anywhere, that's going to be generating heat too. Possibly most of it, if you're doing non-CPU-intensive file serving. So the bean counters would probably *like* their fancy new cooling technology to work with HDDs.

    19. Re:What about the hard drives? by GameboyRMH · · Score: 1

      Even if you want to use SSDs, immersion-cooling them is not worth it. The SSD is then at risk of succumbing to PCB softening like every other component, and for what? Do SSDs even get that hot?

      Network booting is probably the best solution here.

      --
      "When information is power, privacy is freedom" - Jah-Wren Ryel
    20. Re:What about the hard drives? by Anonymous Coward · · Score: 0

      How about any optical networking you would like to do. Have you seen all the rubber covers to keep dust out? I can't imagine that oil has a 0 refraction score.

    21. Re:What about the hard drives? by 1s44c · · Score: 1

      Network booting is probably the best solution here.

      Using SAN disks would work quite nicely too. Either way you still end up with disks somewhere that will need to be cooled outside the oil bath.

      Plus this setup uses at least 4 times the floor space of vertical racks.

      This is a neat idea, but I don't see it working in practise.

    22. Re:What about the hard drives? by Life2Death · · Score: 0

      Why not use SSD?

    23. Re:What about the hard drives? by c6gunner · · Score: 1

      This is why you should just set up shop in an abandoned aquarium. Kick Shamoo out of the tank, fill it with mineral oil, pitch your server racks into it, and you're off and running!

      Of course, all your techs would have to be qualified scuba-divers, but you'd save a fortune on cooling!

    24. Re:What about the hard drives? by Senior+Frac · · Score: 1

      RTFA fail

    25. Re:What about the hard drives? by Anonymous Coward · · Score: 0

      Keep guessing ;)

      Electrolytic capacitors fail by themselves because they dry out. 3 year is average lifespan. Oil will prevent this from happening, possibly extending their lifespan.

      http://en.wikipedia.org/wiki/Electrolytic_capacitor#Electrolyte

      As you can see, the electrolyte is water based. Oil does not mix well with water. Also, there will be very little thermal cycling as the oil will maintain a very nice, stable temperature for these caps.

    26. Re:What about the hard drives? by Anonymous Coward · · Score: 0

      >>I'm racking my brains trying to think of something this would mess up.

      yeah. upgrades.

      Manager: "go upgrade the ram in that machine"
      tech: "but we're all out of arm-gloves"
      Manager: "well I guess you're taking a baby-oil bath today"
      tech: FML.

    27. Re:What about the hard drives? by Compaqt · · Score: 1

      Wha?

      The only thing the actual article mentions about hard drives is: "applying a coating to hard drives".

      Applying a coating is hardly the same thing as encapsulating, and others have pointed out that hard drives aren't air-tight.

      Granted, I later saw something about hard drives in the a page off the home page of the company doing this, but that page is not "the article".

      --
      I'm not a lawyer, but I play one on the Internet. Blog
    28. Re:What about the hard drives? by Glendale2x · · Score: 1

      Hard drives are not air tight, they have breather holes.

      --
      this is my sig
    29. Re:What about the hard drives? by rcamans · · Score: 1

      Racking your brains. heh heh. You had better submerse your brains in mineral oil or the bearings might start to smoke.

      --
      wake up and hold your nose
    30. Re:What about the hard drives? by WhiteDragon · · Score: 1

      Plus this setup uses at least 4 times the floor space of vertical racks.

      This is a neat idea, but I don't see it working in practice.

      There's no reason that has to be true. The racks could in theory be elevated, so you have one horizontal rack above another.

      --
      Did you mount a military-grade, variable-focus MASER on an unlicensed artificial intelligence?
    31. Re:What about the hard drives? by gumbi+west · · Score: 1

      Not sure where you saw anything about spraying. The FAQ says, "each hard drive is encapsulated using GRC’s proprietary hard drive sealing system." They also say SSDs can just go in, and that the grease on the CPU must be replaced with foil.

    32. Re:What about the hard drives? by petermgreen · · Score: 1

      Are you seriously saying you would run a server farm under oil without the cooperation of whoever made the servers?! I'd be far more comfortable swapping the cooling components for watercooling ones than immersing unknown materials in oil.

      --
      note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
    33. Re:What about the hard drives? by Ol+Olsoc · · Score: 1
      I don't know if this is the wave of the future, but I can see a few issues.

      The stuff is sitting in hot mineral oil. Replacing a server that's gone bad is going to be a lot more exciting, ermm, messy. Hard drive problems are the limiting factor, they say that they could run up to 60 degrees C (140 F) if the HD's weren't there. That would be a part of the heat recovery system, and it would be more efficient. So if you do get the HDs external to the servers, then retrieving a bad server will take actual physical protection. The data center of the future might be a little less pleasant to work in than it is today. Oily electronics, hot temperatures, a drastically different workflow. There just might be something else a lot better, I hope.

      --
      The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
  7. Problems? by Anonymous Coward · · Score: 0

    This has been proposed and tried many times, but I seem to recall that it doesn't work well in the long run, because the oil must be nonconductive, but to be nonconductive it has to be pure. Impurities make the coolant conductive. You also can't have hard disks in there, maintenance is a nightmare and I'd imagine that cables in and out of the tank are a bit problematic too.

  8. Re:Mineral oil = nightmare by haxor45 · · Score: 2

    Sorry, the link is supposed to be goatse, but they took down the blog I used for this :-(
    Next time folks, today isn't my day
    Anyway, the post is actually correct, I did read such an article sometime ago.
    Don't remember the link unfortunelly

  9. Boiling water by dimethylxanthine · · Score: 1

    This could make a lot of hot tea for that staff running the DC... Or CRAC. Or spin some turbines perhaps?..

    1. Re:Boiling water by Compaqt · · Score: 2

      Excellent brainstorming ... Farmville servers will replace nuclear plants for the purpose of boiling water to spin turbines. Ingenious. (Saying this half sarcastically, yet some marketroid will actually run with it.)

      --
      I'm not a lawyer, but I play one on the Internet. Blog
  10. It certainly looks cool... by fuzzyfuzzyfungus · · Score: 1

    I have to wonder at their claim that it works well with standard OEM gear. Even most cheap consumer shit monitors the speed of at least the CPU fan and tends to freak out if a fan that is supposed to be there is either absent or performing substantially below expected speed(and, given the relatively high stall current of these fans, burning a trace isn't totally out of the question, if the fan or fan controller isn't smart enough to give up after a short time...) Given that high-density servers and blades will be forced to shut down, or cook, within a short time after fan failure, I'm strongly suspecting that a lot of system management cards and firmwares will flip out at you nonstop if the fans are either removed or acting as mineral oil pumps at a few hundred RPM...

    I'd also be interested to see the details of how they handle the rack rails. One major advantage of horizontal mounting is that, assuming the bearings aren't completely shot, a single person can easily and safely pull out even some 8U monstrosity. Vertically, you are working against gravity, and your spindly geek arms wouldn't need to be all that spindly to encounter some hilariously expensive dropping-the-coolant-lubed-hardware-with-an-expensive-crunch accidents. Unless the system is specced exclusively for dinky little blades and half-depth 1Us(in which case those massively built coolant ponds with aisles on all sides aren't actually all that dense...), they would have to have something in place to make working with the bigger stuff doable.

    1. Re:It certainly looks cool... by slim · · Score: 1

      It seems pretty trivial to replace the fans with mock fans, that either always reports an OK fan speed, or does something with the measured oil temperature.

      True enough about the horizontal mounting and the weight. I don't fancy dealing with a heavy unit dripping with baby oil -- but surely, since they have an installation, they've addressed these practicalities?

    2. Re:It certainly looks cool... by fuzzyfuzzyfungus · · Score: 2

      I imagine that some sort of pneumatic/hydraulic piston system(like the one used on many vertical-open car doors and the like) would be well within the realm of the possible, and offer a zero-apparent-weight slide-up ; but I can't see whether or not there is anything of the sort from the pictures provided.

      The one picture showing "easy serviceability" shows the operator having completely unracked a relatively small server. That doesn't scream 'easy' to me. Pull server up, hold with one hand, unclip a bunch of fiddly cables with the other(don't lose them in the oil pit, or you'll have to give the temp a snorkel...) lay it flat, pop the top? Aw hell no. The whole point of slide rails is that you can get to the guts without having to disconnect any cables(unless connected to the particular bit you are swapping, of course) or unrack anything.

      All blades would be OK, assuming you never need to get at the back of the blade chassis; but I pity the fool who has to disconnect and reconnect(and be sure to get them back in the right places... those 8 are on different VLANS!) a nasty mess of ethernet cables just to swap a PSU or something.

    3. Re:It certainly looks cool... by Cap'nPedro · · Score: 1

      I have to wonder at their claim that it works well with standard OEM gear. Even most cheap consumer shit monitors the speed of at least the CPU fan and tends to freak out if a fan that is supposed to be there is either absent or performing substantially below expected speed

      Enter the BIOS (hit DEL during Power On Self Test), go into the Power or PC Health (depending on what BIOS you have). Alter the value of CPU FAN to "Not Monitored" or "Ignored". Hit F10 (or whatever yor key is) to save settings and reboot.

      SpeedFan (etc) will still give you a speed readout if a fan is connected, but your BIOS won't complain if one isn't.

      This procedure should be similar for UEFI based systems.

    4. Re:It certainly looks cool... by Anonymous Coward · · Score: 0

      Yes. Dripping in oil. Because all the crap we have on our desk already (food, paper, other gadgets) really like to be coated in "mineral oil like coolant". Next we'll find out that the extra ingredient in the mix is carcinogenic or causes your nuts to fall off or whatever. Oh but you aren't supposed to be exposed for more than a few minutes a year because servers have 99.99% uptime. Yeah okay.

      The stereotype is IT guys are a bunch of scrawny dorks. Why is it management has no problem giving us a 80lb server and telling us to mount it on the top rack without any lifting equipment? We really should have a day were we run normal people in the office through a datacentre for an hour. Have all the old kit and an empty rack. Let them lift everything, get it fixed on shelves/rails, try to snake the cables from God knows where you have a spare network port to the server etc. Then coat the badboys in oil and have the biggest guy in the office try to do it. I think the idea would be dropped pretty quick (along with the old piece of junk servers you've been wanting to get out of your datacentre).

    5. Re:It certainly looks cool... by loose+electron · · Score: 1

      "most cheap consumer shit monitors the speed of at least the CPU fan and tends to freak out if a fan that is supposed to be there is either absent or performing substantially below expected speed"

      Got it backwards - Since the Pentium 1, there has been thermal monitor diodes inside the CPU to monitor the silicon temperature. Fan speed was dialed up-down as a function of the temperature.

      Cooling using fluid has been around for many years, This is so 1985.

      Immersion of HDD? Thats a quick crash and burn. Except for specialty devices they are open to acquire external same air pressure, thru a sub-micron filter, yes! HDD are not the primary source of heat in a computer however.

      --
      www.effectiveelectrons.com "chips that work" Analog, RF, Mixed Signal
    6. Re:It certainly looks cool... by fuzzyfuzzyfungus · · Score: 1

      At least on recent stuff, it typically goes both ways: The motherboard has thermal data from the CPU, and adjusts the output on the PWM line going to the fan, which adjusts its speed. At the same time, to detect fan faults, the motherboard monitors the Tach line coming from the fan. If it doesn't like what it sees, some sort of complaining(from an optional alert on the low end, to a compulsory alert that has to be manually cleared at boot, to just plain shutting down) is typically the result.

      As somebody mentioned further up, it wouldn't be rocket surgery to emulate the expected fan signals if needed; but I would expect that a reasonable percentage of management firmwares would at least complain, if not worse, without correctly faked signals.

    7. Re:It certainly looks cool... by DarkOx · · Score: 1

      Some drives equalize pressure with diaphragm rather than being open to any actual gas exchange with the outside. I am not sure these would fair much better though because the oil is much denser than the air in the drive and is going to therefor push the diaphragm in raising the pressure in the drive to the maximum the diaphragm will allow which might very well be outside the range the engineers designing the drive ever expected to encounter

      --
      Repeal the 17th Amendment TODAY! Also Please Read http://www.gnu.org/philosophy/right-to-read.html
    8. Re:It certainly looks cool... by Zencyde · · Score: 1

      I'm already imagining a convenient system. You could have all of the servers submerged with an engine lift to roll around and pick up the servers. One could design the server racks to be serviceable from that situation. But a portable lift is a cheap and easy to use piece of equipment. I'd hope no one is lifting this stuff out with their hands. ;)

      --
      What day is it? Could you please tell me?
  11. Re:Mineral oil = nightmare by Anonymous Coward · · Score: 0

    No soup for you.

  12. Tried this by Anonymous Coward · · Score: 0

    I did this years ago for a high school tech prep course. I used a non-conductive refrigerant, and an old pc. It's a nightmare in the fact that it's almost impossible to seal, and if you want to add any peripherals it's next to impossible. Once everything is coated there is no going back. It is a great cooling solution, but has major drawbacks.

    1. Re:Tried this by Albanach · · Score: 1

      The video shows the tech unplugging and replugging RAM, so I'm assuming there was no need to seal peripherals that don't have moving parts.

  13. Paging Dr. Freeze by upside · · Score: 5, Informative

    1999 Have I been reading Slashdot that long?

    --
    I'm sorry if I haven't offended anyone
    1. Re:Paging Dr. Freeze by RivenAleem · · Score: 1
    2. Re:Paging Dr. Freeze by Ken+Broadfoot · · Score: 1

      Not as long as I.....

      --
      Bitcoin pyramid: Join here: http://www.bitcoinpyramid.com/r/1427 it's FREE!
  14. Re:Mineral oil = nightmare by Anonymous Coward · · Score: 0

    Leakage is definitely a concern, because mineral oil creeps along capillaries. Additionally, it has the nasty tendency to dissolve some plastics over time.
    Of course, with careful parts selection you can work around some of these problems, and it is not as bad for servers as for workstations (no need to deal with mice/keyboard/whatever), but still...

    But worth a try. I just would not bet (much) money on it.

  15. Re:Mineral oil = nightmare by migla · · Score: 1

    That's ok. I can live without it.

    --
    Some of my favourite people are from th US; Vonnegut, Chomsky, Bill Hicks.
  16. Rack density by Askmum · · Score: 2

    The setup looks nice as it is, but having 42" racks laying on their backs never gives the same rack density than the same racks standing upright.

    1. Re:Rack density by Anonymous Coward · · Score: 0

      Course it can. Just stack some racks on top of the horizontal racks.

    2. Re:Rack density by fuzzyfuzzyfungus · · Score: 2

      All you need is very low ceilings, allowing multiple tiers of racks in what would ordinarily be a 1-story room. You can then hire malnourished Dickensian urchins to scuttle, bent over in a permanent half-crouch, among the dripping coolant tanks, swapping drives and cards... Just remember to flog them frequently, to remind them of their place, or you'll find one cooking his fish-and-chips in the hot-coolant line and clogging the heat exchanger with crumbs.

    3. Re:Rack density by Anonymous Coward · · Score: 0

      Or mount half of them upside down to the ceiling

    4. Re:Rack density by Bill+Barth · · Score: 1

      Given that you can lay two of these racks back to back and then run them end to end, and that you can remove most of the regular AC equipment from your room, the amount of stuff you can get in your datacenter is the same.

      --
      Yes...I am a rocket scientist.
    5. Re:Rack density by rjstanford · · Score: 1

      How do you keep their loose pants on in that case? I wouldn't think that the normal spec grubby suspenders would do the trick any more.

      --
      You're special forces then? That's great! I just love your olympics!
    6. Re:Rack density by Enry · · Score: 1

      Normal racks are 15-20kW. If you can fit 100kW into a rack that uses up the floor space of 3 normal racks, you're still ahead of the curve in terms of power usage.

      Then again, I'm getting 4*14*12=672 cores in a single rack using less than 20kW. Unless they're using 150W CPUs, I have no idea how they need to cool 1/10 of a megawatt in a single rack.

    7. Re:Rack density by LWATCDR · · Score: 1

      Why not just use water cooling. Water is a lot more efficient at removing heat than oil. Just install water blocks in place of the heat sinks. You wouldn't need to worry about sealing the contacts and pins which you would with oil. and if done correctly maintenance would be too bad.
      I believe that Cray used immersion cooling on their Cray-2 line using Fluorinert.

      --
      See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
    8. Re:Rack density by Bill+Barth · · Score: 1

      Have you tried fitting a water block in a blade lately? How about 2000 of them? :)

      --
      Yes...I am a rocket scientist.
    9. Re:Rack density by LWATCDR · · Score: 1

      But they are not using blades they are using 1Us. At least that is what it looks like. I find it odd that they are putting the HDs in the oil and in the server. To be as green as possible I would think that having the server boot from the network and having a SAN would be a better choice. And it would beat the daylights out of having to pull an HD out of an oil soaked server.
      How about you used heat tubes from the copper block on the chip connect to an easy to fill water block/ heat exchanger for a blade system.
       

      --
      See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
    10. Re:Rack density by Anonymous Coward · · Score: 0

      That's important. What is the real trade off on watts per square foot. I could imagine 100kW of air cooled containment could be 40 square feet.
      That would be a two foot wide by four foot deep cabinet times 5 cabinets. Granted you would still need a 'cold aisle' for all that air delivery and a plenum for a return but it seems like you would need an aisle for access and a raised ceiling anyways.

  17. Smells like placebo marketing by Anonymous Coward · · Score: 1

    Mineral oil immersion cooling is documented to cause deterioration of components. Perhaps the principals/marketing people are counting on the servers to be obsoleted before they suffer component failure?

  18. Four years of college.... by Ahimoth · · Score: 5, Funny

    and I still get to go home smelling like fries!

  19. Don't hard disks need air inside them? by Anonymous Coward · · Score: 0

    I can understand cooling servers in mineral oil for everything except the hard disk. However, hard disks have a breather hole and actually need air on the inside, because the hard drive heads ride above the disk itself on pressurized air -- among other things this is why hard disks don't work well in space. So what happens when a disk is immersed in oil? The oil MUST get inside of the hard disk due to the breather hole and expansion/contraction of the air inside of the disk as the disk's temperature changes. So I can understand "easy hardware changes" -- but what do you do with a disk that's full of oil to get the oil out? OPEN IT? ?

    1. Re:Don't hard disks need air inside them? by gblackwo · · Score: 1

      I'd like to know the answer to this also. There are harddrives designed for high altitude/low pressure that are apparently completely sealed, but they are of course going to be more expensive.

      You are absolutely right and this is a somewhat normal cause of failure for harddrives. If you use your laptop on a plane and the cabin pressure drops sufficiently, or perhaps you are in Colorado- the read/write head can crash into the platter due to the lack of dense air to ride upon.

    2. Re:Don't hard disks need air inside them? by Larryish · · Score: 1

      The company that is pushing the oil-as-a-coolant solution may need to remake servers in a form that is conducive (conductive? :) to oil cooling.

      Think "oil-cooled server appliance".

  20. False by pubwvj · · Score: 1, Insightful

    This is all based on a false premise.
    There is still heat H to be removed from the server.
    The oil speeds up the conductivity between electronics and medium
    BUT the heat still must be removed and dumped.
    This is not truly more efficient and certainly not "greener" than using air as the medium.

    This will make servicing the equipment more difficult and possibly shorten the life of the equipment as well as requiring more complex coolant systems thus making this system less efficient and "green".

    1. Re:False by RingDev · · Score: 3, Informative

      Having worked in a fair number of server rooms, I'd say that the freequency of needing to service equipment has been dropping significantly over the lat 15 years. These days, it's almost a non-issue. I don't think I've pulled a single server for anything but replacement in the last 4 years.

      Transfering heat to fluids is significantly more efficient, both on the recieving site (in the server) and the giving side (in the cooling tower). It requires less energy to transfer heat from components to the water (ie: no fans or heat sinks). And it requires less energy to transfer heat from the water in the cooling tower (ie: much smaller chiller/AC unit). So it is more efficient. Acording to the article, their solution consums 50% less energy than the traditional air conditioning and fans.

      -Rick

      --
      "Most people in the U.S. wouldn't know they live in a tyrannical state if it walked up and grabbed their junk." - MyFirs
    2. Re:False by Anonymous Coward · · Score: 0

      OK, let me as you this: would you rather your entire body be 36.9C uniformly, or your left eyeball be 150C but the rest of your body be 36.875C? Excess heat still needs to be removed, but that may even be taken care of by thermodynamics by simply opening the windows (on a cool day). Even if that's not enough, dispersing the heat over a large volume (read: area) still helps a lot, otherwise you need the ambient temp to be so low that the tiny processor that is generating the heat can lose it to the air around it, which is an extremely bad conductor, even when it is moving (i.e. you have fans).

      And you should think a little before calling others stupid, lest you turn out to be the stupid one, with the added bonus of being an asshole.

    3. Re:False by gblackwo · · Score: 1

      The greenest solution would be to make the heat useful. Air cooled server rooms often self heat portions of their buildings, and reduce heating costs in the winter.

      An oil cooled system could transfer heat to the hot water in your building via a heat exchanger and lower your gas/electric costs. And this would be a much more efficient process than using your computers to keep your building warm...

    4. Re:False by slim · · Score: 1

      I wonder what the cost savings would be overall. TFA says it pays for itself in 1-3 years, but that's marketing and it's vague.

      If it really saves a load of energy, I can imagine datacentre ops whining about the hassle and the greasy fingers and so forth, saying it's not been worthwhile -- while the suits who commissioned it look at the bottom line of their electricity bill, and deem it well worth a few inconvenienced staff.

    5. Re:False by CarboRobo · · Score: 1

      Er, no it isn't. You still have the same amount of heat, but the oil is far, far more efficient at moving it away than air is.

    6. Re:False by slim · · Score: 1

      (I feel a bit patronising spelling this out)

      It's simple. Pick a component that emits heat. The CPU is a good example. Of course people are working on more efficient CPUs, but they will always generate heat. ... and if that heat stays near the CPU, the temperature will increase until the CPU stops working -- or melts -- or a safety cutoff kicks in.

      So, you have to move that heat away from the CPU, and that in itself takes energy -- powering a fan; powering an air-con unit to keep the room cool enough for the fan to be any use, etc.

      What we have here is a method to move the heat away from the CPU, that uses less energy than the conventional method of blowing air around. ... and if it doesn't work, it's going to have some pretty dissatisfied customers, because the results will show up directly on their energy bill.

    7. Re:False by DrgnDancer · · Score: 1

      Liquids handle heat exchange better. It's the reason your car is water (technically anti-freeze, but you get the point) cooled instead of air cooled. I've never worked with anything this fancy, but we had water cooled racks in the data center of one place I worked. The way it worked was you had a water cooled "radiator" mounted as the back door of the rack. Servers sucked in cool data center air, heated it up normally, shot it out their back ends and forced it through the radiators. There were also fans on the radiators to increase the flow of air through them. When air came out the back, it wasn't appreciably warmer than when it went in. The water was cycled through an external cooling unit. The whole kit and caboodle reduced our power bill by about 20%, with no noticeable difference in system heat. It was going to take some years to pay for itself, but it definitely saved on our monthly bills. The big downside from my point of view was than I lost my "warm aisle" where I could stand to defrost :-)

      Assuming you don't mind the short term expense for long term saving, I'd totally recommend the water cooled racks. They worked, saved money, saved energy, and had minimal impact on the server components or maintenance complexity. I'm not sure that coating everything in baby oil is a good solution. I'd be curious to see what the relative benefit is vs. a less invasive technique like the water cooling we used.

      --
      I don't need a million points of light, just two points of multi-mode fiber and a 10 Gig-E router.
    8. Re:False by netdigger · · Score: 0

      Yes you are correct that heat (H) that is being produced by the server is going to be the same no matter what cooling method is being implemented.

      The traditional cooling medium, air, can not hold as much energy (H) as the cooling fluid used by Green Revolution cooling. To give air better energy absorbing abilities you must first remove energy from it. This in itself requires mass amounts of electricity via a AC system. With oil cooling this ac system is skipped and replaced with a heat exchanger where then the heat is dissipated with a cooling tower.

    9. Re:False by PitaBred · · Score: 1

      And this right here is why America is failing in science. Learn some thermodynamics kid, and pay special attention to the "heat capacity" and "thermal conductivity" portions.

    10. Re:False by petermgreen · · Score: 1

      Electronics are designed to run at temperatures well above normal outdoor air tempreature in most places and so an efficiant cooling system should be able to cool them with nothing more than moving the working fluid arround yet we still end up actively pumping heat using phase change refridgeration to cool air below ambiant. To me this sounds like a MASSIVE WTF.

      --
      note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
  21. Re:Mineral oil = nightmare by Inzite · · Score: 2

    There's a good play-by-play report of a hobbyist's adventures in mineral oil cooling here. The first page is just an introduction, but contains links to all the juicy bits on successive pages.

    Sorry, no goatse.

  22. Maintenance problems by mangu · · Score: 1

    maintenance is a nightmare

    You bet it is! Imagine the mess when you need to replace anything. Not to mention finding the fault. The first thing you do is unplug and plug again the cables just to see if it's just a bad contact problem. Now try to do that when everything is in an oil bath.

    1. Re:Maintenance problems by Compaqt · · Score: 1

      You don't do maintenance. This is for when you have hundreds of servers of the same kind. If it fails, you start up another in its place.

      --
      I'm not a lawyer, but I play one on the Internet. Blog
  23. Aha by jovius · · Score: 1

    So this is what happens when you drool too much.

  24. new line of lead lined accessories for summer by Anonymous Coward · · Score: 0

    from surviveuncomfortably.consume; these garments (the hats are hysterical), buildings, villages etc.., although heavy, & hot as hell, seem to stay put, & deflect stuff. & even to add weight & the ultimate odd atmosphere protection, there's the pyramidal plasmatic material underneath, for after the crud stops flying/the metal melts.

    it's all in the genuine native elders rising bird of prey leadership initiatives, aka teepeeleaks etchings, where it also says that weapons are absolutely unrequired in a living/thriving civilization, after the chosen ones terrifying holycost ends, which is scheduled..

  25. condensation problems... by Jerom · · Score: 2

    When I experimented with mineral oil based cooling, the main issue I had were water droplets condensing on the surface of the cold mineral oil and then promptly sinking... towards the motherboard sitting in the bottom of the old aquarium I was using as a case. Of course there were solutions to this problem but it was a quick and dirty (you can take that dirty word quite literally) test back when I was a student, so we gave up on the idea pretty quickly.

    I wonder how they have managed to solved the condensation problem.

    1. Re:condensation problems... by Phs2501 · · Score: 2

      I wonder how they have managed to solved the condensation problem.

      They run their oil at 40C. If the dew point in your server room is that high you have other problems...

    2. Re:condensation problems... by RingDev · · Score: 1

      If you had condensation (water) that was dense enough to sink in your oil, you were using the wrong type of oil.

      Mineral oil is way more dense than water. And condensation that occurs should sit at the top of the pool and never create an issue (and it should be pretty easy to skim off).

      -Rick

      --
      "Most people in the U.S. wouldn't know they live in a tyrannical state if it walked up and grabbed their junk." - MyFirs
    3. Re:condensation problems... by Jerom · · Score: 1

      Are you sure about the density? According to http://en.wikipedia.org/wiki/Mineral_oil the density of mineral oil is 0.8g/cm3 while that of water is 1gg/cm3.

      J.

    4. Re:condensation problems... by Jerom · · Score: 1

      of course!

      Our oil was much colder: we were using the cooling element from an old refrigerator (we planned on overclocking that old 386 :-)) and managed to get it down to a couple of degrees celsius :-)

    5. Re:condensation problems... by Anonymous Coward · · Score: 0

      A working in the petrochemical process industry (transformer oils and such), I have to say that if you are using oil that is heavier than water, you are using the wrong oil.
      Bitumen, the heavy, nearly solid stuff, has a density about the same as water. The "regular" oils are about 800 g/liter IIRC.

    6. Re:condensation problems... by slim · · Score: 1

      ... and you needn't have. The aim isn't to get the CPU cold. It's just to keep it at operating temperature. Overclocking makes it churn out more heat, which means you need to draw more heat away -- but the point with oil is that it can conduct the heat away more readily. Elsewhere in this thread, it says the system we're talking about here has the oil at 40C. You could have done similar.

      The only reason we have to use air con to keep server rooms so cold, is that air is such a bad thermal conductor.

    7. Re:condensation problems... by RingDev · · Score: 1

      I stand corrected. I was thinking that heavy mineral oil was ~1.1-1.2, but it's only .9. Whoops!

      luckily, it would still be easy to skim, you'd just have to pull off the bottom instead of the top.

      -Rick

      --
      "Most people in the U.S. wouldn't know they live in a tyrannical state if it walked up and grabbed their junk." - MyFirs
    8. Re:condensation problems... by Jerom · · Score: 1

      I understand that... *now*...but back then in the drunk haze that college was, colder == faster, which was silly I know. Besides we used the mineral oil to cool beer too (worked like a charm)

    9. Re:condensation problems... by Anonymous Coward · · Score: 0

      Nope, mineral oil is less dense than water at approx. .8g/cm3. http://en.wikipedia.org/wiki/Mineral_oil The last time I checked, water was almost 1g/cm3 at 40C, so the water would eventually sink to the bottom of the pool.

    10. Re:condensation problems... by asvravi · · Score: 1

      Where did you pull that out from? Mineral oil density is 0.8gm/cc, water is 1gm/cc. Way more?? http://en.wikipedia.org/wiki/Mineral_oil Condensed water as a result indeed does sink to the bottom. An interesting account of a mineral oil cooled PC here - with updates over a period of 3 years! It also discussed the problem of condensed water sinking to the bottom. http://www.pugetsystems.com/submerged.php

    11. Re:condensation problems... by DarkOx · · Score: 1

      Well the easiest solution I can think of would be to have a the electronics suspended off the bottom of the tank, not much maybe an inch or so. You then need a plate a little larger than the electronics mounted over top of them tilted at an angle. It needs to not go the edges of the tank so as to allow the warm oil to rise and slide up the incline on the other side to the top of the tank where it will cool and then roll back down, along the top side. Any water that sinks will also roll down the top side and past the electronics were it will harmlessly collect in the bottom inch of the tank. Periodically when enough water has built up in the bottom of the tank you can siphon the water out stopping the flow slightly before all the water is removed so as not to remove any of the oil.

      That assumes the cooling is passive or if the oil is being mechanically circulated its being down slowly enough not to impact the trajectory that water droplets fall at much or dislocate the water from the tank bottom.

      --
      Repeal the 17th Amendment TODAY! Also Please Read http://www.gnu.org/philosophy/right-to-read.html
    12. Re:condensation problems... by DarkOx · · Score: 1

      The colder the media is the faster it will draw heat away from the component. See Newtons law of Cooling. So yes if you are over-clocking and need to move LOTS of heat away from the tiny surface area of a CPU die, it makes all kinds of sense to want the temperature difference to be as great as possible; provided that does not cause you other problems you can't or don't want to deal with solving like condensation.

      --
      Repeal the 17th Amendment TODAY! Also Please Read http://www.gnu.org/philosophy/right-to-read.html
    13. Re:condensation problems... by Anonymous Coward · · Score: 0

      If you had condensation (water) that was dense enough to sink in your oil, you were using the wrong type of oil.

      Mineral oil is way more dense than water. And condensation that occurs should sit at the top of the pool and never create an issue (and it should be pretty easy to skim off).

      -Rick

      No, that's not right. The specific gravity of mineral oils is in the range of 0.85 - 0.88. Oil floats on water.

    14. Re:condensation problems... by badkarmadayaccount · · Score: 1

      Actually, lowering thermal noise would increase stability at lower voltages, enabling more overclocking.

      --
      I know tobacco is bad for you, so I smoke weed with crack.
  26. There has to be a better way by erroneus · · Score: 1

    Okay, oil is more efficient than air. But the problems with this are plainly obvious when it comes to anything that falls in the area of maintenance and upgrades.

    I wonder what gasses can be used instead of oil? Something that wouldn't likely leave a residue? Substituting a gas for a liquid might reduce some efficiency, but you are still containing the unit completely and entirely. A lot of efficiency can be added merely through the act of containment. There must be some sort of gas that can be pumped into a container that will serve almost as well as oil.

    And is this really so much better than liquid cooling systems placed directly on the electronics? It can't be cheaper.

    1. Re:There has to be a better way by MichaelSmith · · Score: 1

      I wonder what gasses can be used instead of oil?

      Refrigerated air might be best. Cool it before it passes through the system, so it can remove more heat. Of course you either pay for the energy to pre-cool the air, or you locate your data centre somewhere very cold. If thats not good enough you could look at supercharging. Compress the air around the system. That increases density and heat capacity.

    2. Re:There has to be a better way by Anonymous Coward · · Score: 0

      http://www.bit-tech.net/modding/case-mod/2010/09/15/the-mineral-oil-pc-by-andrew-mollman/1

      The biggest problem is the mineral oil damages PVC. There still has to be a radiator of some sort though. The biggest advantage it has over a traditional liquid cooling setup is the connectors of hoses to worry about. And presumably, if your radiator is working well enough, you shouldn't have to worry about condensation.

      As for ease of maintenance, that's really all in how they design the case.

      And no... it is most definitely NOT cheaper.

    3. Re:There has to be a better way by sh3rp4 · · Score: 1

      I use a combination of Nitrogen and Oxygen with misc other gasses added for flavor. Works pretty well with two modified AC units in my server room. Down side is fire (because of the O2) and my IT staff keep dropping dead from breathing the mixture.

      I have to work on that.

    4. Re:There has to be a better way by qubezz · · Score: 1

      Who do you use for IT staff, fish? Since it sounds like your joke describes air, unless the flavor is chlorine gas, the fire risk indicates there is enough oxygen for humans.

    5. Re:There has to be a better way by Anonymous Coward · · Score: 0

      Sulfur Hexafluoride is used in high voltage switchgear and some electronics. It's really thick and heavy: you can see it "pouring", and it slows fans down considerably. Also insulates and cools pretty well. The downside is it's really expensive and your enclosures have to be sealed.

    6. Re:There has to be a better way by Anonymous Coward · · Score: 0

      Supercritical fluid, CO2 perhaps.

  27. Re:Mineral oil = nightmare by smpoole7 · · Score: 2

    > I don't fancy the messy job of making hardware changes though.

    THIS. If it's made by man, it will eventually fail and will require service or replacement.

    The cost in labor (and cleanup!!!) (and replacement oil!) (and trips to the emergency room for employees who slipped and fell in the oil on the floor!) (not to mention the lawsuits) make this a supremely dumb idea. Now add in the cost of the hermetically-sealed rack(s) and it would be difficult to imagine a dumber idea.

    Google is pretty innovative about stuff like this. They use their own in-house version of Linux on commodity hardware, thousands upon thousands of PCs in each data center. But they still use air cooling and air conditioning because, at the end of the day, it's the best bang for the buck. You would save more money by using more efficient equipment, I think -- as you replace each unit, just try to find one that's a little "greener." Or, consolidate servers and use virtualization. Be creative. But DON'T seal the danged thing in a rack filled with oil!

    Now, if someone would design a liquid-cooled rack mount computer with coolant connections, you might could make an argument for that. Run hoses and put the radiators and fans on the side of the building in the shade, maybe. But I still don't see how it would be cost-effective, and the square footage required for the hoses and radiators (or whatever else you plan to use) will hardly improve the expense.

    Not that I'm opposed to liquid cooling, per se. Continental makes a liquid-cooled 30KW FM tube transmitter and it's nice and quiet. My big transmitters still use big tubes and require huge blowers. They're so loud you can't use the phone in the building, and if I'm doing PM and have both the main and auxiliary transmitters on at the same time, it's so loud you can hardly think. As the prices on high-powered solid-state transmitters continue to drop, we're replacing the older tube units. Now THAT'S cost-effective.

    Just my opinion, but if the boss ever told me to fill our racks with mineral oil, I'd take him out to dinner and have a LONG talk with him. (Not that he'd ever propose something so foolish. We have enough trouble with leakage on our hermetically-sealed, oil-filled dummy loads.)

    --
    Cogito, igitur comedam pizza.
  28. Fukucenter by Nobo · · Score: 1

    Wait till the tsunami wipes out the generators.

    1. Re:Fukucenter by Ksevio · · Score: 1

      Then at least when the nuclear servers meltdown, it won't be onto the units below.

    2. Re:Fukucenter by SYSS+Mouse · · Score: 1

      If the tsunami wipes out the generators, there will be no power to run the data center themselves.

      But on the other hand, what if the heat exchange pump breaks down, and the liquid solution ends up boiling.....

    3. Re:Fukucenter by guruevi · · Score: 1

      French fries!

      --
      Custom electronics and digital signage for your business: www.evcircuits.com
  29. subject should not be required by manitee · · Score: 1

    do cooling fans inside the servers need to be disabled? seems like churning that fluid would burn them out.

    also, i dont know about lifting a loaded 8U blade housing vertically...

    --
    Four-digit slashdot ID. Recognize.
    1. Re:subject should not be required by slim · · Score: 2

      Yes, fans are removed. RTFA.

    2. Re:subject should not be required by The+Grim+Reefer2 · · Score: 1

      do cooling fans inside the servers need to be disabled? seems like churning that fluid would burn them out.

      Since the mineral oil will dissipate the heat from the components much better, they will most likely be fine. But will still use electricity unnecessarily, so I would guess they should be removed, or disabled.

  30. Not really by The+Grim+Reefer2 · · Score: 1

    I don't see how this can be good for use w/ a server. It does nothing to increase cooling within the room itself. The server is going to emit the same amount of heat regardless of whether it's air cooled, or mineral oil cooled. The mineral oil will transfer the heat from the components faster, but it will not transfer out of the mineral oil into the air as fast. On a hobbyists computer, it will get shut down, or the load will decrease to almost nothing daily and allow the built up heat to dissipate. On a server that runs constantly, this heat will build continually. If the heat can dissipate out of the mineral oil fast enough to stabilize, then using mineral oil is overkill. If it doesn't, then the server will have to be shut down on a regular basis.

    I would guess that a marginal amount of electricity could be saved due to reduced need for cooling fans. But I think that is trivial in comparison to the potential for leaks and the PITA of swapping hardware, not to mention the additional weight on the floor. Then all of trivial things, like the fumes, than any sticker on the equipment eventually coming off and floating around, slippery floors and components, etc.

    1. Re:Not really by The+Grim+Reefer2 · · Score: 1

      Well, I just went back and watched the video in TFA. They are pumping the oil out to a radiator, or are cooling it somehow. With the added cost of needing to pump mineral oil and cool it, I'm not sure where the savings in electricity is coming from. And all the other problem are still present.

    2. Re:Not really by sandytaru · · Score: 1

      Giant data warehouses, in addition to all the fans for their individual servers, also have enormous air conditioners trying to keep the ambient temperature from hitting 40C. All the hot air that is blow OUT of servers has to go somewhere, and if that somewhere is an enclosed warehouse, things would get hot pretty quickly if the room was not originally designed as a data center. I recall hearing something about "chicken coop" style data centers which have windows to allow for air flow and breezes outside, but that means a custom built building which isn't always an option in a world that likes to retrofit when possible. This setup would eliminate the need for anything more than a standard office air conditioner to keep the humans from sweating.

      --
      Occasionally living proof of the Ballmer peak.
    3. Re:Not really by Anonymous Coward · · Score: 0

      Someone working there commented on this in the other article.

      Since the oil is a lot more efficient at transferring heat it doesn't have to be as cool. They run it at 40 C without problems, and at those temperatures it's easy to passively cool the oil. Just run it through radiators outside and it's cool enough to use again.

    4. Re:Not really by qubezz · · Score: 1

      You don't comprehend how much heat high-density server installations produce. If a 1U processing server uses 200 watts, a 30 space rack of them would use and have a heat output of 6000 watts, which is about what most residential electric furnaces put out. Put four of those in a small room. That uses 200A @ 120v, which, by the way, is more power than most residential power drops can deliver. Now how much air conditioning do you need to keep that room at a constant temperature? The answer is a lot.

      Now you take that heat, transmit it into fluid, and it goes through a big outdoor radiator that is occasionally spritzed with sprinklers for evaporative cooling. You are running a few 500 watt pumps vs running 20,000 watts of AC.

    5. Re:Not really by Anonymous Coward · · Score: 0

      The savings should be obvious, in a datacenter as much energy is used on cooling and ventilation as heating CPUs. Presumably circulating oil through an appropriately sized radiator uses less energy than compressing coolant gas in a heatpump AND circulating air.

      From the video it looked pretty straight forward to swap the hardware, just pull it out of the tank, onto the platform that is specifically designed for the purpose.

      If I was designing something like this, I would submerge the CPU boards on rails (no need for cases), and place the disks in un-submersed thermoblocks that would plumb into the oil cooling lines.

    6. Re:Not really by Anonymous Coward · · Score: 0

      With the added cost of needing to pump mineral oil and cool it, I'm not sure where the savings in electricity is coming from. And all the other problem are still present.

      maybe from the total lack of cooling fans and the better efficency of pumping directly the heat-capturing substance instead of the traditional, indirect, air-conditioning system (as in the machines heat the air, the air heats the compressed gas in the AC, then the unit cools the AC unit's gas)?
      it appears that, since the thermal capacity of the oil is so high, you can just pass it through a radiator outside the building and the heat exchange will be quite easy, even on a warm weather

  31. Re:Mineral oil = nightmare by smpoole7 · · Score: 1

    I honestly wonder if it's an April Fool's joke.

    --
    Cogito, igitur comedam pizza.
  32. Re:Mineral oil = nightmare by PopeRatzo · · Score: 5, Informative

    Leakage is definitely a concern, because mineral oil creeps along capillaries.

    I once drank half a bottle of mineral oil, and let me tell you, leakage was definitely a concern.

    --
    You are welcome on my lawn.
  33. so the whole appenddicks thing is also scammage by Anonymous Coward · · Score: 0

    it was tried. the results are seen here & there, from time to time. imagine the audacity of the neo-gods' 'scientists'? hymens? chariots? honestly... are they still here? that's why even the inherently awesome buttonless monkeys & birds etc..., still do not feel safe? yikes

  34. Re:Mineral oil = nightmare by smpoole7 · · Score: 1

    And one other thing, for any of you hobbyists out there who plan to try this.

    Heat generally rises, but remember that many computers are designed so that the cooling fans force the air *horizontally* across the components. Case in point: my Dell Poweredges. They have a bank of fans near the front that force air "sideways" across the CPU, RAM and other heat-producing parts.

    Unless you put in some sort of coolant pump to circulate that oil (or water, or whatever you plan to use), or *replace* the heatsinks with sealed, carefully-designed systems that pull out the heat and throw it elsewhere, you will probably destroy your computer in the long run.

    In a word: if you try something like this, be careful. Use an infrared thermometer to confirm that you don't have hot spots.

    --
    Cogito, igitur comedam pizza.
  35. Improvement? by yomammamia · · Score: 1

    Would work much better with solid state hard drives.

  36. Single point of FAIL by DominicSayers · · Score: 1

    Four racks sharing one pump? What could possibly go wrong?

    1. Re:Single point of FAIL by slim · · Score: 1

      So have two pumps? I don't suppose they're expensive.

  37. babys welcome Parent Hood Planners by Anonymous Coward · · Score: 0

    the php extension of the unstoppable planet/population rescue kode base is expected to add a more local reconstructible dimension to the 'inevitable' ongoings.

    baal hearings are planned for the royal pains in the butt, but no positive outcome is expected.

  38. definitely not for everyone by SpinningCone · · Score: 1

    it's hard to tell but they might have a special adapter for plugs. i have read that cables will wick the oil in the shielding braid (usb cat5 etc) which can cause a mess.

    also there are a lot of server rooms out there that aren't that organized. i'd imagine in a working installation everything would end up oily, i'd be also a bit wary of installing hard drives in these things. i thought they had the pressure hole for a reason and if your coating failed you could have a massive drive failure.

    that said i'm sure it does a good cooling job and it would be easy to integrate it into the building hot water and heating system.

    overall though I don't think this will make it big

    1. Re:definitely not for everyone by Anonymous Coward · · Score: 0

      it's hard to tell but they might have a special adapter for plugs. i have read that cables will wick the oil in the shielding braid (usb cat5 etc) which can cause a mess.

      also there are a lot of server rooms out there that aren't that organized. i'd imagine in a working installation everything would end up oily, i'd be also a bit wary of installing hard drives in these things. i thought they had the pressure hole for a reason and if your coating failed you could have a massive drive failure.

      that said i'm sure it does a good cooling job and it would be easy to integrate it into the building hot water and heating system.

      overall though I don't think this will make it big

      I work for the company. We haven't seen any wicking.

      We've also worked had to make the room as non oily as possible. It's really not that hard.

      Hard drives are sealed.

    2. Re:definitely not for everyone by Hadlock · · Score: 1

      I noticed in the second video, they only show the process of pulling the computer out. Wiping everything down with an oil-absorbent cloth is suspiciously absent, I noticed. Secondly, there doesn't seem to be a refined system for catching oil drips. I guess once the system is in the oil, it's not coming out but once or twice a year at most, but it's still a flammable liquid that is accumulating inside of an enclosed space. Raised floor systems aren't the easiest to clean up oil spils, I would assume. I'm no clean freak, but running this sort of system without further details on the cleanup leave me very wary.

      --
      moox. for a new generation.
    3. Re:definitely not for everyone by Glendale2x · · Score: 1

      What I noticed is the guy's hands were covered in oil, and when he opens the lid of the server it's resting against his shirt, likely covering it in oil as well. This kind of idea seems like a unholy mess: oil everywhere. I rarely remove my servers once racked, but I do occasionally plug a USB drive or put a DVD in them.

      Oil immersed servers are probably likely to end up in the trash straight away rather than end up on the used market, donated, or reused elsewhere.

      --
      this is my sig
  39. oil in between the card/memory/etc. contacts? by seifried · · Score: 1

    So the oil is not electrically conductive (a good thing right?). What happens when it seeps in between connectors, i.e. into the ram slots or PCIe slots? You start getting really odd random problems, or? How do they address this problem? Also how do you clean the system if you need to service it (i.e. replace bad ram/cards/etc.). If you don't the oils going to get into the slots for memory/cards/etc. when you start swapping components out.

    1. Re:oil in between the card/memory/etc. contacts? by slim · · Score: 1

      My guess about the contacts is that once things are plugged together, they're touching and oil won't break that contact.

      I seriously suspect that the expectation is that a system won't ever be repaired -- if it breaks, it's binned. This is likely to be justifiable based on some sums to do with MTBF, depreciation etc. Of course that kind of thing only really works if your operation scales to hundreds or thousands of machines -- or, I suppose, if an insurance company takes on the spread of that risk.

    2. Re:oil in between the card/memory/etc. contacts? by blueg3 · · Score: 1

      Air is also not electrically conductive and seeps between things much more efficiently than oil.

    3. Re:oil in between the card/memory/etc. contacts? by ProfessionalCookie · · Score: 1

      I had the same problem with air getting in between my contacts.

  40. void(warranty) by SnickleFritz · · Score: 1

    Dell won't service my sever. They say it's too sticky and covered with some sort of slime. :(

  41. Not a new idea... by zevans · · Score: 2

    Cray-2 used Fluorinert. In 1985. Related jokes and memes abounded until... dunno. Certainly they were still part of HPC culture when I started my career in 1994.

    --
    "... and more and more now there are all kinds of electronic goodies available" -- Pink Floyd 1972
    1. Re:Not a new idea... by ajlitt · · Score: 1

      Everything old is new again. In other news, someone just discovered that distributing DC within a rack uses less power.

    2. Re:Not a new idea... by georgesdev · · Score: 1

      by 2020 I suppose someone is going to reinvent the transistor!

  42. Re:Mineral oil = nightmare by DigiShaman · · Score: 1

    With a traditional rack (air cooled), your servers will build up heat in the event of an HVAC failure. But at least you have ample time to start a controlled server shutdown sequence. But with liquid cooling, I would imagine the heat buildup would happen very very quickly in the event of a pump failure. You want to talk about a mini-thermal meltdown, 100kW of heat will do that.

    --
    Life is not for the lazy.
  43. Re:Mineral oil = nightmare by GameboyRMH · · Score: 3, Interesting

    Additionally, it has the nasty tendency to dissolve some plastics over time.

    From what I understand, this has been the main problem with immersion cooling. Mineral oil softens PCBs.

    --
    "When information is power, privacy is freedom" - Jah-Wren Ryel
  44. Mineral Oil is not exactly green by rabun_bike · · Score: 2

    A mineral oil or liquid petroleum is a liquid by-product of the distillation of petroleum to produce gasoline and other petroleum based products from crude oil. And it isn't exactly non-toxic nor non-flammable (see link below). Not to mention all the heavy metals still found in many servers. Inevitably some of those metals will be picked up by the circulating oil so disposal might become an issue as well. Don't get me wrong, I like new ideas that save energy but touting it is "totally green" is skipping a few steps.

    http://www.jtbaker.com/msds/englishhtml/m7700.htm

    1. Re:Mineral Oil is not exactly green by Compaqt · · Score: 1

      It's quite interesting that your link mentions mineral oil as irritant to skin, yet it's sold as "baby oil".

      --
      I'm not a lawyer, but I play one on the Internet. Blog
    2. Re:Mineral Oil is not exactly green by Anonymous Coward · · Score: 0

      I work for the company

      Don't know where you got that MSDS, that isn't the fluid we're using.

      We use fluids out of the cosmetics industry. I promise you that isn't rated semi toxic like that MSDS you post.

      And yes, it is non-flammable, if you look at what is technically defined as flammable. While it is combustible, the official IFC (fire report) we have states that the fire suppression needed is no more than a normal data center.

    3. Re:Mineral Oil is not exactly green by rabun_bike · · Score: 1

      A lot of products and materials you wouldn't think twice about have to be handled properly in large quantities and come with all kinds of warnings. A giant server farm immersed in mineral oil I think would qualify as a large quantity user of mineral oil. Check out the MSDS on caffeine or ethanol for example. http://www.jtbaker.com/msds/englishhtml/c0165.htm http://fscimage.fishersci.com/msds/89308.htm

    4. Re:Mineral Oil is not exactly green by StikyPad · · Score: 1

      Good thing, then, that they're not using mineral oil. Yes, it says mineral oil in the summary, but TFA says "a coolant similar to mineral oil."

    5. Re:Mineral Oil is not exactly green by Anonymous Coward · · Score: 0

      What i don't understand is what is wrong with a modularized water cooling solution. You just have pipes on each server that connects to a common pipe for that rack. Then you just need to bring in a pipe under the racks and connect that up.

      I do designs for high voltage power electronics and this has been the standard for years. We have a standard design & spacing for each module therefore the only difficult part to connect up is to the first module to the inlet pipes.

      The biggest advantage of course is that there is already standard procedures for handling glycol cooling.

    6. Re:Mineral Oil is not exactly green by rabun_bike · · Score: 1

      Thank you. I did read the article and as you pointed out they did not specify what the oil used is but said it is similar to mineral oil as you stated. Therefore, I think it is quite appropriate to assume the chemical structure of the unknown oil used in their design is modeled after mineral oil and hence the appropriate subject of my post.

    7. Re:Mineral Oil is not exactly green by metrometro · · Score: 1

      Given that we run an economy that burns through billions of barrels of petrol a year, finding a use for the stuff that doesn't involve setting the shit on fire for heat seems like a pretty big step forward. And given that the petrol economy is basicly a sunk cost in terms of economic impact, using something that is cheap and available to lower power costs sounds pretty sustainable to me.

      They set out to lower power use. They did that, using a material that is far less impactful than the metals in the servers. Good job.

    8. Re:Mineral Oil is not exactly green by Anonymous Coward · · Score: 0

      I work for GRC.

      That MSDS is not for our coolant, our coolant is similar to mineral oil used in cosmetics. Our coolant is a 0 health on the MSDS, while yours is 2. Way different.

      Also, its not flammable according to the technical definition, but rather combustible. The net of that is that according to the IFC (Fire code) rules, the fire suppression needed is no different than normal data centers.

    9. Re:Mineral Oil is not exactly green by Anonymous Coward · · Score: 0

      Hmmm, the insurance issues of having large open baths of highly flammable oil in your datacentre? Oh, and having 100kW of power running into it too? Some mineral oil has a flashpoint below the temperature some of the semiconductors could reach, particularly if the level had got a bit too low so semiconductors were only partly exposed.

  45. Re:Mineral oil = nightmare by AstroMatt · · Score: 2

    Oil will have a far higher heat capacity than air, so the heating rate after pump failure would be far slower than after an air handler failure.

  46. Re:Mineral oil = nightmare by slim · · Score: 1

    I would imagine -- and someone else has surely done the sums -- that the oil would have enough heat capacity, and be kept at a low enough standard temperature, that it could absorb enough heat without the pump running that a controlled shutdown could be done.

    But it's not beyond the wit of man to have a standby pump.

  47. Canada is the answer! by Anonymous Coward · · Score: 0

    Why not simply move your servers to a place like Iqaluit or Yellowknife? It is pretty cold up there.

    1. Re:Canada is the answer! by slim · · Score: 1

      Well, it's not an entirely stupid idea, and big datacentre operators to like to put themselves in coldish places.

      Even so, Iqualuit gets as high as 25C in summer, which is warmer than my server room, and -40C in winter might bring operational problems of its own.

  48. ssd are to small for a data center may for booting by Joe+The+Dragon · · Score: 1

    ssd are to small for a data center maybe for booting with the data on a SAN but that will give off heat.

  49. Maybe if you're running servers in your basement.. by HockeyPuck · · Score: 1

    Wow... awful idea for 99% of datacenters... Especially those that have ceilings greater than 6ft high.

    Let's see... in all of the pictures the submerged rack is placed on some sort of black grid. I'd bet that if you put this rack on a normal datacenter tile floor and 1 drop of oil got on those tiles, you'd have a nice slip n fall lawsuit on your hands. Besides, the thought of having to stock paper towels and a hazardous spill cleanup kit next to every rack doesn't excite me...

    How many vendors actually support this? If I called up IBM, HP, Cisco, and told them xyz failed and you shipped it back to them for RMA and it was "oily" do you think they'd not bill you for the part?

    How do you deal with fibre connections to the equipment in the rack? I'm sure light travels without any significant dB loss when there's OIL between the optics and the cable...

    Anybody know how this stuff would do in an earthquake? Would the oil slosh out of the containers?

  50. Re:Mineral oil = nightmare by slim · · Score: 1

    Google is pretty innovative about stuff like this. They use their own in-house version of Linux on commodity hardware, thousands upon thousands of PCs in each data center. But they still use air cooling and air conditioning because, at the end of the day, it's the best bang for the buck.

    Not any more. In the early days, as I understand it, Google bought the cheapest desktop PCs they could find, made them netboot, and filled warehouses with them. Their software would deal with hardware failures by routing around the broken server. It worked out cheaper to leave a broken server where it was, than to locate it and repair it or dispose of it. This made sense when Google was a certain size -- big enough to need a big cloud of servers, but too small to invest in custom hardware.

    Nowadays, however, they have boards made in bulk to their own design, which slot into racks of their own design, and they take cooling very seriously -- because to Google, a 1% saving in energy costs represents millions of dollars. As an example, here's a story about Google patenting a novel approach to water cooling.

    I would be *very* surprised if Google hasn't got someone investigating oil-submersion cooling, even if it's not in their production toolkit yet.

  51. Re:Mineral oil = nightmare by Compaqt · · Score: 1

    AFAIK, Google doesn't replace their servers. If it fails, it fails. Hence the viability of this scheme.

    This isn't for co-loc type data centers where you're buying a $20000 dream machine and lovingly installing it in a cage, and coming by to visit it once in a while with gifts (RAM, HDD) in hand.

    It's for set it and forget it cloud-based, commodity data centers.

    --
    I'm not a lawyer, but I play one on the Internet. Blog
  52. High density != Green data centers by EriktheGreen · · Score: 1

    It seems like a lot of people confuse the ability to cram this many servers into a "rack" with an energy efficient, "green" data center.

    The thing is, even though it's about 5x the power density of a "normal" data center, all you're saving is space that more conventional servers would have taken, and maybe gaining a little efficiency in power at the cost of having to maintain all those mineral oil baths. You still have to supply those servers with network connections, potentially also external storage, power management and backups, and most importantly you have to get rid of all that heat. If you're using conventional chiller technology, that's what's taking up the bulk of the space and complexity, not the footprint of the servers themselves.

    100kw of servers will dissipate (depending on model) about 56% of that power as heat, or 56kw, or about the same as 16 tons of chiller capacity. Assuming you're using your reduced server footprint to cram the building full of servers, then you can easily end up needing thousands of tons of cooling capacity, with the attendant cost and complexity of plumbing, plus a backup unit for 2N redundancy.

    But you've saved a few square feet by using old school cooling techniques. Congratulations.

    1. Re:High density != Green data centers by Vegemeister · · Score: 1

      So where does the other 44% of the power go?

  53. Might be good for static installs by DBett · · Score: 1

    But what about the real world of data centers where configurations are constantly changing and new data connections are being made. Seems like the applications are pretty limited.

    1. Re:Might be good for static installs by slim · · Score: 1

      Is that the real world any more? It seems to me that people spending real money on datacentres are virtualising everything -- so once commissioned, hardware won't get touched again until it breaks -- at which point it's disposed of, not repaired. Configuration changes and "new data connections" are managed in software -- VMs, VLANs, that kind of thing.

      I am curious about disposing of equipment that's been used in this way. Is there some solvent bath to clean things up so they don't go to the recyclers coated in oil?

  54. Re:Mineral oil = nightmare by AchilleTalon · · Score: 1

    Mod this up, it's hilarious!

    --
    Achille Talon
    Hop!
  55. Re:Mineral oil = nightmare by Lifyre · · Score: 1

    The oil holds 1200 times as much heat as air so it would be MUCH slower to heat up...

    --
    I'll meet you at the intersection of "Should be" and "Reality"
  56. Re:Mineral oil = nightmare by AchilleTalon · · Score: 1
    To be precise: heat capacity of air is 1004 J/kgK, oil is 2000 J/kgK and water is 4180 J/kgK.

    So, heat capacity of oil is twice air's heat capacity.

    --
    Achille Talon
    Hop!
  57. Re:Mineral oil = nightmare by Lifyre · · Score: 1

    Actually since the mineral oil is so thin it's fairly trivial to deal with. It makes it more complex to pull the hardware and bring it to a test station because you need to account for dripping oil but other than that there isn't nearly as big a difference or problem as you seem to believe.

    --
    I'll meet you at the intersection of "Should be" and "Reality"
  58. Marine engines by Kupfernigk · · Score: 1

    What do you expect would be the effect of circulating oil failure on marine engines with electric oil pumps (which many of them have)? Yet such a failure is very, very rare. Hint: Duplex pumps, standby emergency generators

    --
    From scarped cliff or quarried stone she cries "A thousand types are gone, I care for nothing, no not one."
  59. Re:Mineral oil = nightmare by slim · · Score: 1

    There's also thermal conductivity to consider. Oil may have twice the heat capacity of air, but it will also spread that heat throughout itself much faster.

    Air: 0.025 W/(mK)
    Mineral oil 0.138 W/(mK)
    Water: 0.6 W/(mK)

    Of course the key property of mineral oil is that it has better thermal conductivity than air, while still being an electrical insulator.

  60. Seems like a bad idea by Anonymous Coward · · Score: 0

    First, the racks are now on their side and in a sealed container - you've just lost a lot of real estate in your machine room (namely the space from 3 feet to 7 feet up).

    Also seems like submerging the entire unit in oil is going to cause problems. Most components were never designed to operate under those conditions, which means that you are now running your mission critical servers under conditions that the hardware was never designed/tested for.

    The vast majority of heat in a modern computer is generated by the CPU - liquid based CPU coolers are well established technology that does not subject the system unusual conditions (such as immersion) that would handle probably 80% to 90% of the heat generation. Also most rack-mount chassis are designed with air circulation in mind. Not at all clear that oil would circulate in the ways necessary to bleed off the heat.

    Can you imagine having to service a computer which has been sitting in a pool of oil? It will drip and seep everywhere when you try to work on it, and the oil will get into the cables and connectors and interfere with the electrical contacts. Not to mention that you will need gloves and other protective clothing to avoid being covered in oil yourself.

    Just seems like a bad idea.

  61. herp derp oil by Anonymous Coward · · Score: 0

    Modern systems were designed with the assumption that air would be the cooling medium. Taking something designed for AIR and putting it in oil is not a solution to anything but the length of your hardware warranty. If you took a bird and put it in oil, it wouldn't work very well anymore now would it?

  62. Re:Mineral oil = nightmare by Nadaka · · Score: 1

    With oils convection, thermal conductivity and heat capacity, these are the reasons to choose submersion over air for cooling. When it takes a thousand times more energy to raise a cubic centimeter of oil by a degree, you are in a much more forgiving situation than with air.

  63. Re:Mineral oil = nightmare by EndlessNameless · · Score: 1

    Does anyone use a single HVAC unit? If your datacenter cannot sustain the failure of a single HVAC unit, you need to invest in your infrastructure a little more.

    The same would apply to submersion-cooled equipment. Redundant pumps with sufficient cooling towers to tolerate failures.

    The failure of a single part should never bring anything down unless the part that fails is "the building".

    --

    ---
    According to the latest ruleset, this post should be modded as Vorpal Flamebait +5.
  64. Re:Mineral oil = nightmare by pixelpusher220 · · Score: 1

    Actually no it wouldn't. Because oil/liquid distributes heat much more efficiently than air, when the pumps or fans aren't running the system is left with only the natural properties of the medium it is in for cooling.

    Air is bad, hence why you have directional fans in cases to force air onto the CPU and other chips within the cases.

    Liquids are great heat conductors so the heat will be pulled away from the components much faster. So the heat load is distributed faster, keeping the individual components (which is what 'fails') cooler than if they were left in open air.

    To be fair, the HVAC failure is not the same as the pump failure. Pumps are more akin to the fans inside the cases since they are moving stuff through cases. In the end though, oil/liquid is much better for heat distribution and when left to fend for themselves, components will survive longer in uncooled non-moving oil than even moving but non-cooled air.

    --
    People in cars cause accidents....accidents in cars cause people :-D
  65. Re:Mineral oil = nightmare by yakatz · · Score: 1

    With a traditional rack (air cooled), your servers will build up heat in the event of an HVAC failure. But at least you have ample time to start a controlled server shutdown sequence. But with liquid cooling, I would imagine...

    No need to imagine. Do some calculations and maybe even some prototyping and see what happens.
    Verbum Sap.

  66. Re:Mineral oil = nightmare by Life2Death · · Score: 0

    It softens stools, but not copper coated in fiberglass.
    From our hacking projects, its also a bitch and two halves (haha) to clean off of parts. Dishwashers are the only real hope to RMA something...

  67. Re:Mineral oil = nightmare by Anonymous Coward · · Score: 0

    This. Have a backup and a backup for the backup.

    My workplace had a problem recently when one of two server room HVAC units went down and management refused to do anything about it because the office was supposed to relocate anyway and nobody wanted to sink money into the existing site. However the office still hosted many mission-critical servers.

    The remaining HVAC unit took on the load once shared with the broken unit. This situation held for a few weeks. Then the remaining unit died leaving the server room with no cooling.

    Servers do not like reaching internal temps of well over 100F. They turn off.

    I did not have to explain anything to the customers. Somebody else had to do that. I gather most of them were not very understanding.

  68. Re:Mineral oil = nightmare by TheLink · · Score: 2

    Those figures don't give the full picture.

    1 kg of air takes up a lot more space than 1 kilogram of oil.

    So the heat capacity of oil in a server is far more than twice the heat capacity of air in a server.

    --
  69. leaking oil by Anonymous Coward · · Score: 0

    One thing I don't understand about these mineral oils: They're supposed not to be conductive right? So what if mineral oil leaks in between connectors? Ok, I guess processor and memory sockets are pretty tight, but what if you'd ever need to add more memory to a server? The memory socket will already be soaked with mineral oil. You'd have to completely clean it before the new memory could be installed. And how would you do that?

  70. Re:Mineral oil = nightmare by UnknowingFool · · Score: 1

    It is often recommended in industrial applications that if you have a pump that is critical, you should have 3 of them for one purpose. One that is operational. One on standby to be switched on immediately if the first one fails. The last one is a backup. Also the pumps should be rotated into operation and maintained. In the real world this is not practiced especially if costs are being trimmed.

    --
    Well, there's spam egg sausage and spam, that's not got much spam in it.
  71. I've touched the TACC unit by Takionbrst · · Score: 1

    It was kind of funny, I got to tour the TACC through a lab contact, and the most interesting thing wasn't Ranger (the fastest publicly owned supercomputer in the world), but this odd looking unit they have in the back room pushed up against a wall. Walking up to it, we thought it was filled with flourinert-- but then the systems manager stuck his finger in and licked the liquid off, explaining it was mineral oil! It's pretty amazing, they cool this thing using a swamp cooler. Just a pump with a heat exchanger that feeds a water loop through an exterior wall, to an evaporator. And apparently this thing works well during the Austin summers-- not as bad as Houston, but those of us who've had to endure it know that it can get pretty humid.

  72. Re:Mineral oil = nightmare by petteyg359 · · Score: 0

    If it's made by man, it will eventually fail and will require service or replacement.

    So hire the monkeys and their koala and prairie dog minions to make it for us, so it won't fail.

  73. Re:Mineral oil = nightmare by S.O.B. · · Score: 1

    Awesome...you made my day.

    --
    Some of what I say is fact, some is conjecture, the rest I'm just blowing out my ass...you guess.
  74. watts/sq ft by Anonymous Coward · · Score: 0

    This product doesn't seem that competitive with air flow cabinets on a power density basis...

    Imagine: 100kw's jammed into say 8 cabinets at 12.5kw each. That's 2ft wide x 4ft deep x 8 cabinets ~ 64 sq feet

    That image seemed to be about the same size just for that secondary containment, never mind all that coolant piping and valves.

  75. Arrrgh, the mess! by docilespelunker · · Score: 1

    Being a HV engineer, I know the mess that oil can make. It's not fun, especially if you're sent in to fix something! Imagine in a datacentre full of air - to swap something in the middle you - have a coffee - wonder over - replace the thing - have another coffee. In one filled with oil, you - look at the floor in dissmay - shut the whole centre down - drain all the oil (into a tank the same size as the data centre) - vent the atmosphere for several hours - put on protective gear - wade about in whats left of the oil - replace the thing - climb out - have 4 showers and still smell of oil - fill the centre back up with oil and finally turn it back on. Basically, with air you get coffee twice. With oil, you get no coffee. Not good, not good at all!

  76. Re:Mineral oil = nightmare by DigiShaman · · Score: 1

    So what your saying is this. If I throw off 100kW of power in a tank full of oil, it will not get hot enough to fry fish and chips?

    --
    Life is not for the lazy.
  77. Been done by Anonymous Coward · · Score: 0

    Aside from the cray comment, A friend of mine did this 20 years ago, but it wasn't because of cooling reasons, it was because it looked cool. He chose mineral oil because it was neither conductive nor corrosive.

  78. Re:Mineral oil = nightmare by pixelpusher220 · · Score: 1

    No I'm saying it will take significantly longer to get that hot than if you had the same failure of an HVAC system.

    If you experience such an outage in a data center, time is one thing you definitely want a good amount of.

    --
    People in cars cause accidents....accidents in cars cause people :-D
  79. Warranties? by Anonymous Coward · · Score: 0

    First thing I thought of was having to RMA a product covered in oil. Something tells me these hardware manufacturers are going to love to point at this cooling solution as the problem in many cases or simply have an issues with components covered in oil.

  80. Re:Mineral oil = nightmare by Amouth · · Score: 1

    But it's not beyond the wit of man to have a standby pump.

    Or 26 of them - and yes i know a place where this happened.. and no they only needed 2 pumps operational to carry the load .

    --
    '...if only "Jumping to a Conclusion" was an event in the Olympics.'
  81. magnetic fluid movement by tumaru · · Score: 1

    Would it be possible to use magnetic fluid to move it through this new system? All you would need it some magnetic material in the fluid and a bunch of really small magnetic pumps.

  82. addressing the wrong problem by Gravis+Zero · · Score: 0

    the problem is that there is too much heat which mostly from the processors. the processors generate heat be cause of the amount of power they consume and retain. a better designed CPU could solve this problem and in fact it's already been made. ARM processors put of very little heat and the next generation off them is going to be faster than 2GHz with four cores.

    fix the problem dont just treat the symptoms.

    --
    Anons need not reply. Questions end with a question mark.
  83. Re:Mineral oil = nightmare by BillX · · Score: 1

    and once it does, you can use it to make popcorn to eat while watching the meltdown happen :-) With air, not much to do but open all the windows and mutter "oh shit" repeatedly.

    --
    Caveat Emptor is not a business model.
  84. 3M has an ongoing project on immersion cooling by Anonymous Coward · · Score: 0

    They use Novec Engineered Fluids. In the case of the link below, they use Novec 7000, a low-boiler. The liquid evaporates, then condenses, back into the vat. It's basically a 2-phase cooling system. Novec fluids are inert, dielectric fluid. It doesn't make a mess like mineral oil do. And it's environmentally safe, too.

    Features of Novec 7000:
      Low GWP (370, 100-year ITH)
      Excellent dielectric properties in event of leakage or other failure, will not damage electronic equipment
      Zero ozone depletion potential (ODP)
      Good materials compatibility
      Low toxicity
      Nonflammable
      Non-corrosive
      Good thermal stability
      Useful at extreme low temperatures
    Viscosity is less than 20 cSt at -120C

    http://www.facebook.com/3MNovecdatacentercooling

  85. Re:Mineral oil = nightmare by Lifyre · · Score: 1

    You could probably use the heated oil for making the popcorn too... But honestly if you can't shutdown your system in the amount of time it takes to heat up that oil then you are doing something seriously wrong.

    --
    I'll meet you at the intersection of "Should be" and "Reality"
  86. Re:Mineral oil = nightmare by toddestan · · Score: 1

    On the other hand, if you're considering a pump failure versus an air conditioner failure, you have to consider that with an A/C failure the individual fans in each of the servers will keep moving air through the systems, which will keep them relatively cool until the ambient air temperature in the room gets too warm. With a pump failure, presumably the oil in all the servers will immediately stop moving which would probably lead to them all overheating faster than with the air cooled servers. I guess the lesson would be to have multiple redundant pumps.