Slashdot Mirror
Startup's Submerged Servers Could Cut Cooling Costs
1sockchuck writes "Are data center operators ready to abandon hot and cold aisles and submerge their servers? An Austin startup says its liquid cooling enclosure can cool high-density server installations for a fraction of the cost of air cooling in traditional data centers. Submersion cooling using mineral oil isn't new, dating back to the use of Fluorinert in the Cray 2. The new startup, Green Revolution Cooling, says its first installation will be at the Texas Advanced Computing Center (also home to the Ranger supercomputer). The company launched at SC09 along with a competing liquid cooling play, the Iceotope cooling bags."
someone urinates in the cooling liquid, that is.
Until you have to try and RMA that CPU :)
the new Xeon 5600's run at less power than previous CPU's. and SSD's also run a lot cooler. how much does this liquid cooling enclosure cost and what is the performance compared to just upgrading your hardware?
HP is going to ship their Xeon 5600 servers starting on the 29th
..computers, allow me to label this a "fad"
The idea is funky, but to get good cooling you want convection (every joule of pump energy from a circulating pump gets transferred into the oil at yet more heat) which means deep tanks which means, to the server environment, goodbye high density.
The ONLY thing that has changed since I was doing this is the affordability of SSDs, which mean that now it is practical to immerse the whole computer, and the mass storage too, which makes things a lot simpler and cheaper, and means you really can be JUST oil cooled, not oil cooled mainly, except for air cooled HDs etc.
TOP TIP from an old hand.
If you are going to oil cool by immersion, buy the latest top quality hardware, because once immersed it stays there, you'll only pull it once to see why it sucks.
BIGGEST mistake experimenters make is using old hardware, cos you always end up playing with it, making mess, ahh fsckit..
Nota Bene if you are building one of these in anger, make allowances for the significant increase in the weight that the oil makes.
HTH etc
http://slashdot.org/~GuyFawkes/journal
How much harder does it make doing standard move cables/switch harddrives/change components maintenance?
One of the advantages of a standard rack to me is that all of that is fairly easy and simple, so you can fix things quickly when something goes wrong.
'Sensible' is a curse word.
Here's the ease of service video.
http://www.youtube.com/watch?v=-q0sTFX1DFM
It was in order to build a totally silent computer, the cooling aspect worked OK, nothing spectacular, not of you layout the case properly, buy fans with decent blade profiles and proper bearings, and decent aftermarket heatsinks, but the total silence was beautiful... even ATX PSU's do make a noise, you only notice when you immerse *everything*
http://slashdot.org/~GuyFawkes/journal
The place where oil immersion makes sense is in the data center, especially at a large one where the surrounding buildings already have a chilled water loop for cooling. All you need is a heat exchanger to turn hot oil into cold, and cold water into warm. You don't have to turn over building fulls of air.
Was I the only one who read the headline and immediately thought of some kind of under water data center. That would have been cool!
You'll obviously need to be scaling before you invest in a system that involves a big vat full of oil.
Also, what does the fire marshall think of a big vat full of oil? Hazardous disposal? Oh boy... some company goes BK, and they leave behind a big vat full oil and outdated electronics.
I didn't dig deep enough to see if they are actively pumping the oil or not. If they are, they're not doing it right. Any system that really cuts cooling costs should be using a LTD engine to transform the heat into useful work.
Of course, you still need to reject the heat someplace. At one place, it was my understanding that they had a helluva time trying to explain to some manager why they had to cut a hole in the building to let heat out of the server room. It's the same basic thermodynamics of "what happens if you leave the refrigerator door open". The room just gets hotter.
So. You'll have to have some kind of oil-air heat exchanger someplace. The hole for an oil line coming out of the server room is smaller... but it's an oil line. Back to the hazard factor...
Don't get me wrong. I understand why they used oil in things like Crays. The rate of heat exchange between the electronics and the oil is evidently better. It's the same reason why 50 degree water gives you hypothermia in 10 minutes and 50 degree air doesn't.
So. That leads us to the questions: Is your overall system efficiency going to be better in some way by running hotter? Does that savings offset the cost of the oil system?
Plainly, a commodity Intel server box doesn't run hot enough to require oil for effective heat transfer, unless you overclock it. If you can get twice the effective computing power in a room with fire-hot overclocked servers and the fancy oil cooler, ok maybe it's worth it?
Note: I don't lay any claim to be an expert in this field. These are just the kind of questions I think a generally intelligent person should ask. If somebody who really knows this stuff can *politely* rebut, then great.
For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
I'm starting a pool. How much longer before the mainframe is re-invented to power cloud computing. I'm taking 1.5 years. Any other bets?
Why not use a water heat exchanger outside the case to cool the oil (while keeping water away from system components, and getting full contact with the entire system)? The water could then go into a loop to cool it. Other coolants could also be used, although water is great from a heat capacity standpoint.
Since the water doesn't touch anything important, it can be dumped into a cooling tower/etc.
To cool one system I doubt it is worth all the trouble, but for a datacenter I bet you could make it very efficient. It is a lot easier to run pipes of water than sufficient ductwork for A/C.
Component replacements could be a pain, unless the rack made it really easy to drain a given case.
There are other ways to make data center cooling more efficient, such as hot aisle containment and individual rack-top coolers blowing cold air directly in front of the racks. There's no reason a modern data center needs to move entire buildings full of air anymore, even without liquid cooling.
Oil immersion may or may not be more efficient, but it doesn't seem like it would scale well. In a large data center where some hardware component is failing on a daily basis, because you have tens of thousands of servers, keeping all that oil contained within the enclosures would be a major challenge. During maintenance, that stuff is going to be getting all over everything, including the tech, who can easily spread it all over anything he touches before he gets around to cleaning up. You'd need a cleaning crew out on the floor constantly.
Hard disks aren't sealed, there's always (at least, on the dozens of disks I've taken apart) a little felt-pad or sticker covered vent on them. I figured it was for equalisation or something crazy, but I'm not positive.
Given hard disks aren't sealed, wouldn't they fill with fluid and assuming they'd still function with a liquid screwing up the head mechanism (given modern disk's head's float above the platter surface on a cushion of air) wouldn't the increased viscosity slow down seek events?
the datacenter in the movie the matrix the humans are emerged in. Good solution too when the "components" fail. Just flush it out the drain ;-)
Won't these servers bathed in oil still have the same thermal output? I don't understand why it would be cheaper to cool oil than it would air or any other medium..
I went to eat some animal crackers and the box said, "Do not eat if seal is broken." I opened the box and sure enough..
"modern data center [...] Oil immersion may or may not be more efficient, but it doesn't seem like it would scale well. In a large data center where some hardware component is failing on a daily basis, because you have tens of thousands of servers"
In a modern, large datacenter you don't repair each failing component; you just redrive your computing load around it.
Oh..... there's something Google didn't think of and try.
-- I was raised on the command line, bitch
Astute move. They're named "Green Revolution Cooling". Everyone knows you can't go wrong when you go "green".
Fuck systemd. Fuck Redhat. Fuck Soylent, too. Wait, scratch the last one.
How do you build a server 'in anger'?
"linux is just DOS with a UNIX like syntax" -- Galactic Dominator (944134)
A server with this Intel Atom equipped mobo draws something like 25-35W under full load. And the performance of these D510 dual core processors is comparable to better Pentium 4 processors.
If somebody who really knows this stuff can *politely* rebut, then great.
Politely? You must be new here, you niggerdick lovin' cocksucker. XD
Can anyone tell me why we cannot simply move many of these servers to northern Canada? Canada has great fibre optic infastructure and the average temperature 8 months out of the year is well below a nominal temperature for cpu's. Blow the cold air in and push the warm air into administrative buildings. Cheap and Green.
Cue tech in scuba gear swimming down through the oil to change a power supply.
Interested in a Flash-based MAME front end? Visit mame.danzbb.com
...you want convection (every joule of pump energy from a circulating pump gets transferred into the oil at yet more heat) which means deep tanks which means, to the server environment, goodbye high density.
Really? You could say the same about air moved by a fan (that the fan's energy contributes to the overall heat). I'm no expert in this area, but I've seen liquid cooled PCs and the only big component is the radiator. I would think you could pack liquid cooled components more densely than air cooled, and you could put the radiator in another room.
Prov 9:8 Do not rebuke mockers or they will hate you; rebuke the wise and they will love you.
I seem to remember mainframes using distilled water for cooling decades ago. Not being a member of the correct priesthood, I was not allowed in the mainframe room, so I don't know how it was set up then. I have seen how oil-filled systems work, and I would hate to work on one. Nasty mess.
Why, without your clothes, you're naked, Miss Dudley!
Cue tech in scuba gear swimming down through the oil to change a power supply.
Would this be a good application for a robot?
I was wondering if it would cut cooling costs to use the building's main incoming water service as a cool heat sink. The part of this that goes to the hot water heater could be used to soak up heat from servers, then passed to the hot water heater which would then have an easier job. Only using the incoming supply to the water heater would avoid problems with warm tap water but in some cases that wouldn't matter (do most people care if the cold taps in the bathroom produce warm water?). If you could use the whole cold supply it would be a bigger heat sink.
These days if your company is underwater your servers might as well be too.
Just curious, and you seem like the guy to ask, has anyone done full center immersion? With the proliferation of shipping container rack systems, would it be possible to seal the entire container into one giant unit with a manhole on the top, then drop in a diver with either tanks or a line and let them do maintenance without worries of spillage? You'd be able to keep the same density as is currently used, since you'd be able to use the normal maintenance space as space for convection currents and the normal A/C units as heat exchangers. If the depth specifically is an issue, you could always move the racks to floor, since a diver wouldn't require a floor to walk on...
Like I said, I'm just curious. I don't really know much about any of this.
No trespassing. Violators will be shot. Survivors will be shot again.
You can submerge traditional platter type hard drives too! We have an enclosure to package the drives before they get submerged.
We've also tried high density. Works great! Actually, this is the golden solution for high density markets. Ever tried to put 12 physical CPU's on one motherboard? Wouldn't happen without liquid. Just so happens it's cheap with our technology.
Our technology raises a lot of questions. We were at last year's Supercomputing conference (10,000 attendee's) and had tons of questions. I would like to believe we had answers...
-Christiaan Best (GRC employee)
I've been pondering that for a while, personally, I just don't see how to create an effective heatpipe from processor or machine to water pipe.
Sounds like a job for Mike Rowe.
It might also be more efficient to use chilled water in pipes to cool the servers directly rather than chilling air and blowing that around a big building.
Especially when it's free. I used to work at a medical center with a big data center. Cold city water was run first to the data center, heat-pumped to a cold-air Liebert, and then the slightly-warmer water was piped on to all the places where cold water is used. A degree or two warmer is quite fine at the tap.
Smart downtown-City data centers would work a deal with the city to do the same thing and stop paying for coal-generated electricity. Maybe the next crop of data centers should be build next to the water treatment plants.
This kind of "green" will be of the "backs" kind.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
Are SSD's submersible?
Good-bye
The Cray 2 had a three stage cooling system; the flourinert was pumped through a heat exchanger and dumped it's heat into chilled water, which was either provided by the site's existing HVAC infrastructure or (more likely, since the dissipation was in the Megawatt range) by a dedicated freon-based water chiller. The 5th generation Cray Inc (as opposed to CCC) also used immersion cooling in a similar vein. Many other Cray machines (YMP, C90 and so on used the same 3-stage cooling system, but the modules weren't immersed in the flourinert, rather the coolant flowed through channels in a thermally conductive plate sandwiched between the two boards of each processor or memory module. This wasn't a means of cooling the boards more cheaply; this was ECL logic... in those days it was the only way you could deliver the required power and have the thing not literally melt.
You are making the assumption that individual servers or even racks must be changed out regularly. Considering how many datacenters are no longer space constrained, but rather power constrained by the number of KW/square foot, or cooling restrained due to local regulations or power consumption, other approaches are valid.
The opposite conclusion is a containerized datacenter/rack cluster using oil immersion as the internal primary coolant, hooking into a datacenter fed cold water heat exchanger mounted at the end of the container. With that you would nominally design for a specific rated giggflop/Gbps for the container as a whole. At first, you have more than that, but as devices fail, you fail in place. When the container performance drops below rated, you swap out the whole container. Considering the depreciation and rated lifecycle/lifetime of servers, this is not an unreasonable proposition. Say, expected rated lifetime of 3 years. Swapping containers with the container manufacturer as a part of a trade-in/leased pool financing plan. When the container is brought in for maintenance, the equipment and personnel necessary to deal with oil immersion are available. The manufacturer can refurbish/replace the servers within to return to the lease pool, or if the conclusion is that it isn't cost effective to do so, drain the bastard and sell off the remaining servers to recycle the container or simply sell off the old container whole as a "below rated" or EoL product.
As a dedicated server lessee I would be interested to know if there is a risk of liquid escaping and destroying hardware, or the system failing and hardware overheating and then failing.
A youtube animation of using evaporative cooling for server cooling.
The fluorinated fluids are low viscosity and evaporate quickly and thus dry allowing quick and easy servicing.
I suppose if this business plan doesn't pan out, we shouldn't say that they went under?
Or, perhaps, if they sign a bad mortgage on their offices, and wind up owing more than the building is work, they'll *really* be underwater.
Thank you, I'll be here all week. Try the veal.
-Z
Why in hells name would you ever think that you'd need to oil submerge things in a stack?
Just stair-step waterfall the damn things. Sure the racks are different, but at least all the machines are easy to get to.
As you said, easier to run water through a building than oil. Compared to oil, it has relatively low viscosity, but also much larger cooling capacity than air. Ground water is usually cool enough to chill computers.
In the servers, you could have heat pipes run to the back, where a large heat sink provides a thermal connection to the chilled water system. That way you could pull them easily and would avoid the mess inside the servers and the costs of the scuba gear.
If the servers are still too hot, have water cooled fins on the side of the rack. You could have chilled water circulate on five sides of the server and still have them easy to pull, while made of only standard components and a couple heat pipes. If all major heat producers were facing the chassis (instead of pointing inwards like it is now), you might not even need heat pipes.
The chilled water gets regenerated by heating large shrimp tanks
Fluorinert is not mineral oil, nor even very similar to mineral oil.
-ring-
Hi, I'd like a high-performance computer submerged in oil.
Would you like fries with that?
Think of all the displaced IT grunts that suddenly gained job experience.
Yeah... you're right:
Have the ROBOT tell the tech to put on the scuba gear and go change a power supply.
GRC's Mark Tlapak tells me that Iceotope's system is "beautiful but costly", while Iceotope's Peter Hopton dismisses GRC as "fishtank manufacturers".
Basically, it looks like a simple solution (a bath) versus a more complex one (individual sealed blades). The discussion is here at eWEEK Europe UK.
Peter Judge UK Editor, eWEEK Europe
this was ECL logic
And there I was thinking they went straight from TTL logic to CMOS logic logic.
The article describes a system where servers are stored in what is essentially a rack laid down on the ground and filled with oil. Now, this is going to be too heavy, I would have thought, to be able to support any off the ground, so you're limited to only using the bottom 60cm or so of each room in your datacenter for server storage. Isn't this going to mean you only get half as many servers in there?
Yes. Works for SSDs, kills HDDs.
If there's gonna be swimming in oil, I demand to be promoted to supervisor of the hot hardware replacement babes.
I interviewed these guys at SC09 for Linux Magazine. There are some close up shots of the servers in the oil.
HPC for Primates. Read Cluster Monkey
Is it at all feasible to run a computer submerged in distilled water? You'd have to ensure that the water remains pure, obviously, but this might be easier than dealing with computers submerged in oil. The obvious advantage is that distilled water is more benign and MUCH easier to work with. Any spills can be cleaned up with a rag, for one thing.
Switch back to Slashdot's D1 system.
Going "green" should be about using and specially wasting less energy, not about ways to better dismiss high energy usage...
"Video bona proboque; deteriora sequor." -- Ovid
Nice. I have no mod points but I did laugh.
My Xbox Live Gamer Card
These guys have really put a lot of thought into the system. The oil is much more efficient at moving heat out of the boxes than air, and holds more heat.
...Then my job here is done...