Slashdot Mirror
A Closer Look At Immersion Cooling For the Data Center
1sockchuck writes "Want to save money on data center cooling? Tip your racks on their side, fill them with mineral oil, and submerge your servers. Austin startup Green Revoluton Cooling first profiled here) has a video demo of its immersion cooling solution, which it says can handle racks using up to 100kW of power. A photo gallery on the company web site shows some early installations."
The thing here is they are commercializing a cooling technique usually reserved for the hobbyist. I don't know about the energy saving claims, but their setup looks fairly organized. Interesting turn for a still niche cooling solution.
Eat sleep die
Your link doesn't work, but I imagine it was a hobbyist. It looks as if this lot have built an industrial-strength product. It's clearly not practical in the home.
The entire computer is submerged in a bath of oil, and the oil is circulated through a cooling tower. I doubt there are any holes for IO below the oil level of the bath, so leakage isn't a concern.
I don't fancy the messy job of making hardware changes though.
Well, on the other hand, if they're supposed to be air-tight, I guess they're baby oil-tight, too.
But there's got to be something or another that doesn't react well with mineral oil, right?
I guess this means they save on fans, and the power to run fans. That's additional power and heat savings right there.
OK, I've got it: what about the CD/DVD drives? Or is it all network IPL in data centers? I'm racking my brains trying to think of something this would mess up.
I'm not a lawyer, but I play one on the Internet. Blog
Sorry, the link is supposed to be goatse, but they took down the blog I used for this :-(
Next time folks, today isn't my day
Anyway, the post is actually correct, I did read such an article sometime ago.
Don't remember the link unfortunelly
This could make a lot of hot tea for that staff running the DC... Or CRAC. Or spin some turbines perhaps?..
I have to wonder at their claim that it works well with standard OEM gear. Even most cheap consumer shit monitors the speed of at least the CPU fan and tends to freak out if a fan that is supposed to be there is either absent or performing substantially below expected speed(and, given the relatively high stall current of these fans, burning a trace isn't totally out of the question, if the fan or fan controller isn't smart enough to give up after a short time...) Given that high-density servers and blades will be forced to shut down, or cook, within a short time after fan failure, I'm strongly suspecting that a lot of system management cards and firmwares will flip out at you nonstop if the fans are either removed or acting as mineral oil pumps at a few hundred RPM...
I'd also be interested to see the details of how they handle the rack rails. One major advantage of horizontal mounting is that, assuming the bearings aren't completely shot, a single person can easily and safely pull out even some 8U monstrosity. Vertically, you are working against gravity, and your spindly geek arms wouldn't need to be all that spindly to encounter some hilariously expensive dropping-the-coolant-lubed-hardware-with-an-expensive-crunch accidents. Unless the system is specced exclusively for dinky little blades and half-depth 1Us(in which case those massively built coolant ponds with aisles on all sides aren't actually all that dense...), they would have to have something in place to make working with the bigger stuff doable.
1999 Have I been reading Slashdot that long?
I'm sorry if I haven't offended anyone
That's ok. I can live without it.
Some of my favourite people are from th US; Vonnegut, Chomsky, Bill Hicks.
The setup looks nice as it is, but having 42" racks laying on their backs never gives the same rack density than the same racks standing upright.
Mineral oil immersion cooling is documented to cause deterioration of components. Perhaps the principals/marketing people are counting on the servers to be obsoleted before they suffer component failure?
and I still get to go home smelling like fries!
This is all based on a false premise.
There is still heat H to be removed from the server.
The oil speeds up the conductivity between electronics and medium
BUT the heat still must be removed and dumped.
This is not truly more efficient and certainly not "greener" than using air as the medium.
This will make servicing the equipment more difficult and possibly shorten the life of the equipment as well as requiring more complex coolant systems thus making this system less efficient and "green".
I would expect oil to be far more efficient than air. It has a hugely greater thermal capacity (hundreds of times), so it can extract much more heat from the chips and similarly hand it over better to the cooling vanes. You use thermal paste to connect the chip to heat sinks better than air - this is a larger scale version of the same thing, where the whole system is immersed in a sort of thermal paste.
Consciousness is an illusion caused by an excess of self consciousness.
There's a good play-by-play report of a hobbyist's adventures in mineral oil cooling here. The first page is just an introduction, but contains links to all the juicy bits on successive pages.
Sorry, no goatse.
maintenance is a nightmare
You bet it is! Imagine the mess when you need to replace anything. Not to mention finding the fault. The first thing you do is unplug and plug again the cables just to see if it's just a bad contact problem. Now try to do that when everything is in an oil bath.
So this is what happens when you drool too much.
When I experimented with mineral oil based cooling, the main issue I had were water droplets condensing on the surface of the cold mineral oil and then promptly sinking... towards the motherboard sitting in the bottom of the old aquarium I was using as a case. Of course there were solutions to this problem but it was a quick and dirty (you can take that dirty word quite literally) test back when I was a student, so we gave up on the idea pretty quickly.
I wonder how they have managed to solved the condensation problem.
Okay, oil is more efficient than air. But the problems with this are plainly obvious when it comes to anything that falls in the area of maintenance and upgrades.
I wonder what gasses can be used instead of oil? Something that wouldn't likely leave a residue? Substituting a gas for a liquid might reduce some efficiency, but you are still containing the unit completely and entirely. A lot of efficiency can be added merely through the act of containment. There must be some sort of gas that can be pumped into a container that will serve almost as well as oil.
And is this really so much better than liquid cooling systems placed directly on the electronics? It can't be cheaper.
> I don't fancy the messy job of making hardware changes though.
THIS. If it's made by man, it will eventually fail and will require service or replacement.
The cost in labor (and cleanup!!!) (and replacement oil!) (and trips to the emergency room for employees who slipped and fell in the oil on the floor!) (not to mention the lawsuits) make this a supremely dumb idea. Now add in the cost of the hermetically-sealed rack(s) and it would be difficult to imagine a dumber idea.
Google is pretty innovative about stuff like this. They use their own in-house version of Linux on commodity hardware, thousands upon thousands of PCs in each data center. But they still use air cooling and air conditioning because, at the end of the day, it's the best bang for the buck. You would save more money by using more efficient equipment, I think -- as you replace each unit, just try to find one that's a little "greener." Or, consolidate servers and use virtualization. Be creative. But DON'T seal the danged thing in a rack filled with oil!
Now, if someone would design a liquid-cooled rack mount computer with coolant connections, you might could make an argument for that. Run hoses and put the radiators and fans on the side of the building in the shade, maybe. But I still don't see how it would be cost-effective, and the square footage required for the hoses and radiators (or whatever else you plan to use) will hardly improve the expense.
Not that I'm opposed to liquid cooling, per se. Continental makes a liquid-cooled 30KW FM tube transmitter and it's nice and quiet. My big transmitters still use big tubes and require huge blowers. They're so loud you can't use the phone in the building, and if I'm doing PM and have both the main and auxiliary transmitters on at the same time, it's so loud you can hardly think. As the prices on high-powered solid-state transmitters continue to drop, we're replacing the older tube units. Now THAT'S cost-effective.
Just my opinion, but if the boss ever told me to fill our racks with mineral oil, I'd take him out to dinner and have a LONG talk with him. (Not that he'd ever propose something so foolish. We have enough trouble with leakage on our hermetically-sealed, oil-filled dummy loads.)
Cogito, igitur comedam pizza.
Wait till the tsunami wipes out the generators.
do cooling fans inside the servers need to be disabled? seems like churning that fluid would burn them out.
also, i dont know about lifting a loaded 8U blade housing vertically...
Four-digit slashdot ID. Recognize.
I'd like to know the answer to this also. There are harddrives designed for high altitude/low pressure that are apparently completely sealed, but they are of course going to be more expensive.
You are absolutely right and this is a somewhat normal cause of failure for harddrives. If you use your laptop on a plane and the cabin pressure drops sufficiently, or perhaps you are in Colorado- the read/write head can crash into the platter due to the lack of dense air to ride upon.
I don't see how this can be good for use w/ a server. It does nothing to increase cooling within the room itself. The server is going to emit the same amount of heat regardless of whether it's air cooled, or mineral oil cooled. The mineral oil will transfer the heat from the components faster, but it will not transfer out of the mineral oil into the air as fast. On a hobbyists computer, it will get shut down, or the load will decrease to almost nothing daily and allow the built up heat to dissipate. On a server that runs constantly, this heat will build continually. If the heat can dissipate out of the mineral oil fast enough to stabilize, then using mineral oil is overkill. If it doesn't, then the server will have to be shut down on a regular basis.
I would guess that a marginal amount of electricity could be saved due to reduced need for cooling fans. But I think that is trivial in comparison to the potential for leaks and the PITA of swapping hardware, not to mention the additional weight on the floor. Then all of trivial things, like the fumes, than any sticker on the equipment eventually coming off and floating around, slippery floors and components, etc.
I honestly wonder if it's an April Fool's joke.
Cogito, igitur comedam pizza.
I once drank half a bottle of mineral oil, and let me tell you, leakage was definitely a concern.
You are welcome on my lawn.
And one other thing, for any of you hobbyists out there who plan to try this.
Heat generally rises, but remember that many computers are designed so that the cooling fans force the air *horizontally* across the components. Case in point: my Dell Poweredges. They have a bank of fans near the front that force air "sideways" across the CPU, RAM and other heat-producing parts.
Unless you put in some sort of coolant pump to circulate that oil (or water, or whatever you plan to use), or *replace* the heatsinks with sealed, carefully-designed systems that pull out the heat and throw it elsewhere, you will probably destroy your computer in the long run.
In a word: if you try something like this, be careful. Use an infrared thermometer to confirm that you don't have hot spots.
Cogito, igitur comedam pizza.
Would work much better with solid state hard drives.
Four racks sharing one pump? What could possibly go wrong?
You're right, it takes a lot of energy to heat up that much oil, that's the point. It doesn't take much energy to heat up a cubic foot of air, it takes a lot more to heat up a cubic foot of oil, the effort here is to remove as much of the energy that would go into melting chips as is possible.
Liquid to air heat exchangers can be made as big as you need pretty cheaply, it's easier to pump 800 cubic feet of air through a big radiator cooling one cubic foot of oil than it is to pump 800 cubic feet of air through an OEM heat-sink.
it's hard to tell but they might have a special adapter for plugs. i have read that cables will wick the oil in the shielding braid (usb cat5 etc) which can cause a mess.
also there are a lot of server rooms out there that aren't that organized. i'd imagine in a working installation everything would end up oily, i'd be also a bit wary of installing hard drives in these things. i thought they had the pressure hole for a reason and if your coating failed you could have a massive drive failure.
that said i'm sure it does a good cooling job and it would be easy to integrate it into the building hot water and heating system.
overall though I don't think this will make it big
So the oil is not electrically conductive (a good thing right?). What happens when it seeps in between connectors, i.e. into the ram slots or PCIe slots? You start getting really odd random problems, or? How do they address this problem? Also how do you clean the system if you need to service it (i.e. replace bad ram/cards/etc.). If you don't the oils going to get into the slots for memory/cards/etc. when you start swapping components out.
Dell won't service my sever. They say it's too sticky and covered with some sort of slime. :(
Cray-2 used Fluorinert. In 1985. Related jokes and memes abounded until... dunno. Certainly they were still part of HPC culture when I started my career in 1994.
"... and more and more now there are all kinds of electronic goodies available" -- Pink Floyd 1972
The company that is pushing the oil-as-a-coolant solution may need to remake servers in a form that is conducive (conductive? :) to oil cooling.
Think "oil-cooled server appliance".
With a traditional rack (air cooled), your servers will build up heat in the event of an HVAC failure. But at least you have ample time to start a controlled server shutdown sequence. But with liquid cooling, I would imagine the heat buildup would happen very very quickly in the event of a pump failure. You want to talk about a mini-thermal meltdown, 100kW of heat will do that.
Life is not for the lazy.
Additionally, it has the nasty tendency to dissolve some plastics over time.
From what I understand, this has been the main problem with immersion cooling. Mineral oil softens PCBs.
"When information is power, privacy is freedom" - Jah-Wren Ryel
A mineral oil or liquid petroleum is a liquid by-product of the distillation of petroleum to produce gasoline and other petroleum based products from crude oil. And it isn't exactly non-toxic nor non-flammable (see link below). Not to mention all the heavy metals still found in many servers. Inevitably some of those metals will be picked up by the circulating oil so disposal might become an issue as well. Don't get me wrong, I like new ideas that save energy but touting it is "totally green" is skipping a few steps.
http://www.jtbaker.com/msds/englishhtml/m7700.htm
Oil will have a far higher heat capacity than air, so the heating rate after pump failure would be far slower than after an air handler failure.
I would imagine -- and someone else has surely done the sums -- that the oil would have enough heat capacity, and be kept at a low enough standard temperature, that it could absorb enough heat without the pump running that a controlled shutdown could be done.
But it's not beyond the wit of man to have a standby pump.
ssd are to small for a data center maybe for booting with the data on a SAN but that will give off heat.
The video shows the tech unplugging and replugging RAM, so I'm assuming there was no need to seal peripherals that don't have moving parts.
Wow... awful idea for 99% of datacenters... Especially those that have ceilings greater than 6ft high.
Let's see... in all of the pictures the submerged rack is placed on some sort of black grid. I'd bet that if you put this rack on a normal datacenter tile floor and 1 drop of oil got on those tiles, you'd have a nice slip n fall lawsuit on your hands. Besides, the thought of having to stock paper towels and a hazardous spill cleanup kit next to every rack doesn't excite me...
How many vendors actually support this? If I called up IBM, HP, Cisco, and told them xyz failed and you shipped it back to them for RMA and it was "oily" do you think they'd not bill you for the part?
How do you deal with fibre connections to the equipment in the rack? I'm sure light travels without any significant dB loss when there's OIL between the optics and the cable...
Anybody know how this stuff would do in an earthquake? Would the oil slosh out of the containers?
Google is pretty innovative about stuff like this. They use their own in-house version of Linux on commodity hardware, thousands upon thousands of PCs in each data center. But they still use air cooling and air conditioning because, at the end of the day, it's the best bang for the buck.
Not any more. In the early days, as I understand it, Google bought the cheapest desktop PCs they could find, made them netboot, and filled warehouses with them. Their software would deal with hardware failures by routing around the broken server. It worked out cheaper to leave a broken server where it was, than to locate it and repair it or dispose of it. This made sense when Google was a certain size -- big enough to need a big cloud of servers, but too small to invest in custom hardware.
Nowadays, however, they have boards made in bulk to their own design, which slot into racks of their own design, and they take cooling very seriously -- because to Google, a 1% saving in energy costs represents millions of dollars. As an example, here's a story about Google patenting a novel approach to water cooling.
I would be *very* surprised if Google hasn't got someone investigating oil-submersion cooling, even if it's not in their production toolkit yet.
AFAIK, Google doesn't replace their servers. If it fails, it fails. Hence the viability of this scheme.
This isn't for co-loc type data centers where you're buying a $20000 dream machine and lovingly installing it in a cage, and coming by to visit it once in a while with gifts (RAM, HDD) in hand.
It's for set it and forget it cloud-based, commodity data centers.
I'm not a lawyer, but I play one on the Internet. Blog
It seems like a lot of people confuse the ability to cram this many servers into a "rack" with an energy efficient, "green" data center.
The thing is, even though it's about 5x the power density of a "normal" data center, all you're saving is space that more conventional servers would have taken, and maybe gaining a little efficiency in power at the cost of having to maintain all those mineral oil baths. You still have to supply those servers with network connections, potentially also external storage, power management and backups, and most importantly you have to get rid of all that heat. If you're using conventional chiller technology, that's what's taking up the bulk of the space and complexity, not the footprint of the servers themselves.
100kw of servers will dissipate (depending on model) about 56% of that power as heat, or 56kw, or about the same as 16 tons of chiller capacity. Assuming you're using your reduced server footprint to cram the building full of servers, then you can easily end up needing thousands of tons of cooling capacity, with the attendant cost and complexity of plumbing, plus a backup unit for 2N redundancy.
But you've saved a few square feet by using old school cooling techniques. Congratulations.
But what about the real world of data centers where configurations are constantly changing and new data connections are being made. Seems like the applications are pretty limited.
Mod this up, it's hilarious!
Achille Talon
Hop!
Well, it's not an entirely stupid idea, and big datacentre operators to like to put themselves in coldish places.
Even so, Iqualuit gets as high as 25C in summer, which is warmer than my server room, and -40C in winter might bring operational problems of its own.
The oil holds 1200 times as much heat as air so it would be MUCH slower to heat up...
I'll meet you at the intersection of "Should be" and "Reality"
So, heat capacity of oil is twice air's heat capacity.
Achille Talon
Hop!
Actually since the mineral oil is so thin it's fairly trivial to deal with. It makes it more complex to pull the hardware and bring it to a test station because you need to account for dripping oil but other than that there isn't nearly as big a difference or problem as you seem to believe.
I'll meet you at the intersection of "Should be" and "Reality"
What do you expect would be the effect of circulating oil failure on marine engines with electric oil pumps (which many of them have)? Yet such a failure is very, very rare. Hint: Duplex pumps, standby emergency generators
From scarped cliff or quarried stone she cries "A thousand types are gone, I care for nothing, no not one."
There's also thermal conductivity to consider. Oil may have twice the heat capacity of air, but it will also spread that heat throughout itself much faster.
Air: 0.025 W/(mK)
Mineral oil 0.138 W/(mK)
Water: 0.6 W/(mK)
Of course the key property of mineral oil is that it has better thermal conductivity than air, while still being an electrical insulator.
With oils convection, thermal conductivity and heat capacity, these are the reasons to choose submersion over air for cooling. When it takes a thousand times more energy to raise a cubic centimeter of oil by a degree, you are in a much more forgiving situation than with air.
Does anyone use a single HVAC unit? If your datacenter cannot sustain the failure of a single HVAC unit, you need to invest in your infrastructure a little more.
The same would apply to submersion-cooled equipment. Redundant pumps with sufficient cooling towers to tolerate failures.
The failure of a single part should never bring anything down unless the part that fails is "the building".
---
According to the latest ruleset, this post should be modded as Vorpal Flamebait +5.
Actually no it wouldn't. Because oil/liquid distributes heat much more efficiently than air, when the pumps or fans aren't running the system is left with only the natural properties of the medium it is in for cooling.
Air is bad, hence why you have directional fans in cases to force air onto the CPU and other chips within the cases.
Liquids are great heat conductors so the heat will be pulled away from the components much faster. So the heat load is distributed faster, keeping the individual components (which is what 'fails') cooler than if they were left in open air.
To be fair, the HVAC failure is not the same as the pump failure. Pumps are more akin to the fans inside the cases since they are moving stuff through cases. In the end though, oil/liquid is much better for heat distribution and when left to fend for themselves, components will survive longer in uncooled non-moving oil than even moving but non-cooled air.
People in cars cause accidents....accidents in cars cause people
From my experience building a hobby oil cooled pc .. I can imagine oil doing a much better job.
Oil has a much greater heat capacity over air. What this translates to is the oil takes a _long_ time to heat up. In my 21 gallon rig, with no cooling or circulation... the oil just sitting still takes several hours to heat up to a dangerous temperature. Try turning off your PC fans and see how long the box lasts!
The inverse is of course true as well.. once it heats up.. it stays hot for just as long.
If you had an efficient way of keeping that oil cool and flowing at a decent rate... I can imagine this working quite well.
For the curious, my oil rig worked great until the HSF popped off the processor one day (probably a combination of weird forces on the (very large) heat sink and maybe a cheat retention clip)... then it didn't. :(
With a traditional rack (air cooled), your servers will build up heat in the event of an HVAC failure. But at least you have ample time to start a controlled server shutdown sequence. But with liquid cooling, I would imagine...
No need to imagine. Do some calculations and maybe even some prototyping and see what happens.
Verbum Sap.
Those figures don't give the full picture.
1 kg of air takes up a lot more space than 1 kilogram of oil.
So the heat capacity of oil in a server is far more than twice the heat capacity of air in a server.
It is often recommended in industrial applications that if you have a pump that is critical, you should have 3 of them for one purpose. One that is operational. One on standby to be switched on immediately if the first one fails. The last one is a backup. Also the pumps should be rotated into operation and maintained. In the real world this is not practiced especially if costs are being trimmed.
Well, there's spam egg sausage and spam, that's not got much spam in it.
It was kind of funny, I got to tour the TACC through a lab contact, and the most interesting thing wasn't Ranger (the fastest publicly owned supercomputer in the world), but this odd looking unit they have in the back room pushed up against a wall. Walking up to it, we thought it was filled with flourinert-- but then the systems manager stuck his finger in and licked the liquid off, explaining it was mineral oil! It's pretty amazing, they cool this thing using a swamp cooler. Just a pump with a heat exchanger that feeds a water loop through an exterior wall, to an evaporator. And apparently this thing works well during the Austin summers-- not as bad as Houston, but those of us who've had to endure it know that it can get pretty humid.
Awesome...you made my day.
Some of what I say is fact, some is conjecture, the rest I'm just blowing out my ass...you guess.
Being a HV engineer, I know the mess that oil can make. It's not fun, especially if you're sent in to fix something! Imagine in a datacentre full of air - to swap something in the middle you - have a coffee - wonder over - replace the thing - have another coffee. In one filled with oil, you - look at the floor in dissmay - shut the whole centre down - drain all the oil (into a tank the same size as the data centre) - vent the atmosphere for several hours - put on protective gear - wade about in whats left of the oil - replace the thing - climb out - have 4 showers and still smell of oil - fill the centre back up with oil and finally turn it back on. Basically, with air you get coffee twice. With oil, you get no coffee. Not good, not good at all!
So what your saying is this. If I throw off 100kW of power in a tank full of oil, it will not get hot enough to fry fish and chips?
Life is not for the lazy.
No I'm saying it will take significantly longer to get that hot than if you had the same failure of an HVAC system.
If you experience such an outage in a data center, time is one thing you definitely want a good amount of.
People in cars cause accidents....accidents in cars cause people
But it's not beyond the wit of man to have a standby pump.
Or 26 of them - and yes i know a place where this happened.. and no they only needed 2 pumps operational to carry the load .
'...if only "Jumping to a Conclusion" was an event in the Olympics.'
Would it be possible to use magnetic fluid to move it through this new system? All you would need it some magnetic material in the fluid and a bunch of really small magnetic pumps.
and once it does, you can use it to make popcorn to eat while watching the meltdown happen :-) With air, not much to do but open all the windows and mutter "oh shit" repeatedly.
Caveat Emptor is not a business model.
You could probably use the heated oil for making the popcorn too... But honestly if you can't shutdown your system in the amount of time it takes to heat up that oil then you are doing something seriously wrong.
I'll meet you at the intersection of "Should be" and "Reality"
On the other hand, if you're considering a pump failure versus an air conditioner failure, you have to consider that with an A/C failure the individual fans in each of the servers will keep moving air through the systems, which will keep them relatively cool until the ambient air temperature in the room gets too warm. With a pump failure, presumably the oil in all the servers will immediately stop moving which would probably lead to them all overheating faster than with the air cooled servers. I guess the lesson would be to have multiple redundant pumps.