Slashdot Mirror
Cooling Bags Could Cut Server Cooling Costs By 93%
judgecorp writes "UK company Iceotope has launched liquid-cooling technology which it says surpasses what can be done with water or air-cooling and can cut data centre cooling costs by up to 93 percent. Announced at Supercomputing 2009 in Portland, Oregon, the 'modular Liquid-Immersion Cooled Server' technology wraps each server in a cool-bag-like device, which cools components inside a server, rather than cooling the whole data centre, or even a traditional 'hot aisle.' Earlier this year, IBM predicted that in ten years all data centre servers might be water-cooled." Adds reader 1sockchuck, "The Hot Aisle has additional photos and diagrams of the new system."
That's really nifty, and I'm sure it works ok and everything, but... how much does it cost?
Just because I can hook a shark from a boat, I do no offer to wrestle it in the water.
For some reason, the filters at work won't let me view the article. Does it happen to mention how much the upfront cost for these bags are?
Living With a Nerd
TFA mentions using the excess heat to heat the building. I wonder how feasible it would be to actually recycle the heat to generate more power? Anyone have an idea on how much heat could be generated by your typical server farm?
"Before God we are all equally wise - and equally foolish"
Albert Einstein
Seriously. What do we do when a RAM module or a backplane fails? Will a simple hardware swap become a task for those trained in hazmat handling? I do not want to be on the help desk when someone calls and says "Help! The servers are leaking!"
I judt got a nre Kinesis keybiartf so please excusr ant egregiou typos.
Won't this cause accessibility issues for the administrators who have to support these servers? Additionally, Google's evidence supports the idea that warmer temperatures are better for the life of some components, such as hard drives. Last, this may work well for traditional servers, but I fail to see how this can be made to support a large SAN array or something similar.
Super cool! ^_^ If they made those for laptops, I'd be all over it. My wife likes to use her HP as a lap warmer, with a blanket... But there I go thinking again... --Stak
Holy happy hippy crap!
we all know what happens when you mix water and server rooms http://www.youtube.com/watch?v=1M_QTBENR1Q better call up Noah
Grandma would be proud of her cold compress technology.
End anonymous moderation and posting on
Interviewer: "Well Mr. Robinson, while your resume is quite impressive, however, you just don't have everything we're looking for to fill the opening on our server maintenance team."
Mr. Robinson: "What do you mean? I have a Masters in Computer science, A+, MCSE, CCNA, CISSP, and 23 years of relevant experience. What am I missing??"
Interviewer: "You see, we're running that new server cooling technology you might of seen on slashdot. I didn't see anything about being SCUBA certified on your resume."
The problem with all this is you need a good piping and plumbing system in place, complete with quick release valves to ensure you can disconnect or connect hardware without having to do a whole bunch piping and water routing in the process. Part of the beauty of racks is you just slide in the computer, screw it in, and plug in the plugs at the back and you're done.
I'm not saying it's impossible, but just building a new case, or blade, or whatever isn't going to do it - you need a new rack system with built in pipes and pumps, and probably a data center with even more plumbing with outlets at the appropriate places to supply each rack with water. This is no small task for trying to retrofit an existing data center.
Not to mention that you have to make sure you have enough pressure to ensure each server is supplied water from the 'source', you cannot just daisy chain computers because the water would get hotter and hotter the further down the chain you go. This means a dual piping system (one for 'cool or room temperature' water and one for 'hot' water). And it means adjusting the pressure to each rack depending on how many computers are in it and such.
The issues of water cooling a data center go WAY beyond the case, which is why nobody has really done it yet - sure, the cost savings are potentially huge, but it's a LOT more complicated that sticking a bunch of servers with fans in racks that can move around and such, and then turning on the A/C. And there is a lot less room for error (as someone else mentioned, what if a leak occurs? or a plumbing joint fails, or whatever. Hell, if a pump fails you could be out a whole rack!).
The ES/9000 that I had contact with was a series of cabinets that were all water-cooled from the outside in...it was a maze of copper pipes all around the edges and back and looked like a fridge. When you opened a cabinet, you could feel a blast of cold air hit you.
It was no trivial feat to do this, they had to install a separate water tank, some generators (I remember one of the operations guys pointing to a Detroit Diesel generator outside in the alley and saying it was just for the computer's water system), moved a bathroom (only water they wanted around the computer was the special chilled stuff), and I can distinctly remember seeing the manuals(!)... 3-inch thick binders with the IBM logo on them, and all they were for was the planning and maintenance of the water system.
No wonder it took almost a year to install the machine.
I work with particle accelerators that draw enough power that we don't have much choice but to use water cooling, and even though we have major radiation sources, high voltage running across the entire place, liquid helium cooled magnets, high power klystrons that feed microwaves to the accelerator cavities etc... the only thing that typically requires me to place an emergency call during a night shift is still water leaks.
Water is just that much of a hassle around electronics. Even an absolutely minor leak can raise the humidity in a place you really don't want humidity, it evaporates and then condenses on the colder parts of the system where even a single drop can cause a short circuit and fry some piece of equipment. After it absorbs dirt and dust from the surroundings it starts attacking most materials corrosively, which may not be noticed at first but gives sudden unexpected problems after a few years. If you don't keep the cooling system itself in perfect condition valves and taps will start corroding and you get blockages. Maintenance is a pain because you have to power everything down if you want to move just 1 pipe etc...
I just don't see why you would go through the hassle with water cooling unless you actually have to, and quite frankly if your servers draw enough power to force you to use water for cooling then you're doing something weird.
Reminds me of the sapphire fire suppression just applied all the time. Or the sealed mineral oil boxes people seem to put computers in. The system could be huge if they apply it right and it actually realizes a 93% reduction in energy cost(I have my doubts). The largest issue I have heard is that it is tricky, but not impossible, to move the heat away from the components once they heat up the liquid.
Hmm... The Cray-2 was cooled via complete immersion in Fluorinert way back in circa 1988. I was an admin on one (Ya, I'm old). So, this is a bit different, but certainly not ground-breaking.
It must have been something you assimilated. . . .
Sixteen years ago, at the end of my highschool career, I was very into overclocking (had multiple celeron 300A). With peltier cooling I was able to run a 300mhz CPU at 450mhz with rock solid stability (ran things like prime95 24hrs a day for weeks). People were starting to experiment with liquid cooling commodity white-box computers.
One of the more interesting applications I saw was an old styrofoam cooler converted into a PC case. All components were submerged in a bath of cold mineral oil. I remember thinking that the data centers of the future would require SCUBA certified technicians in dry suits to swim down to the racks and swap out the broken module. Maybe I was thinking too grand, and this would be feasible as submerge-modules with aquarium like tanks instead of racks.
"Be prepared, son. That's my motto. Be prepared." --Joe Hallenbeck
Look at the cross section photo. This dispenses completely with convection (air flow) and instead designs the system for direct physical contact from the heat sink to the components. Then the water flows behind the heat sink to take the heat away from that.
The problem is that means that you have to make a heat sink with varying height "fingers" on it to meet every component that produces heat (which is all of them), which means every time you change a component you have to redo the heat sink. And of course if you change the motherboard you also have to. With components available from multiple sources (second sourcing) and changing spec mid-model for cost-reduction, you can expect the profile of the heat sink to change frequently during the life of a model. And of course, you probably need to put heat sink goop on a lot of components, that might make enough surface tension that you'd have trouble getting it apart to service it.
Although this is workable, it seems unlikely it would ever be cost-effective. It'd probably be smarter to have certain major (heat-producing) components cooled by direct contact and a plenum for the rest that uses convection to get heat to a radiator-like assembly on the heat sink (except it isn't radiating here, it's absorbing heat).
I think water cooling is likely for servers in the future. Even end-to-end water heat exchange to the atmosphere like this proposes, instead of transferring the heat to the room air and then taking it out with air handlers might be the future. But I'm not sure these guys have the right strategy at the bottom level.
http://lkml.org/lkml/2005/8/20/95
How much does a rack full of water-cooled blades weigh?
Never thought I'd see the UPS become the lightest thing in the server room.
With all those layers it doesnt seem that sliding one of these out and quickly swapping some RAM or any other part is
going to happen.
As well do these Iceotope guys actually make server hardware or just the cooling specs. Who do they get there guts from or are they just advertising and hoping the guys like HP, IBM or SUN (well maybe not SUN) decide to design there next generation of servers with this in mind?
I'd like to see how easy it is for replacement. doesnt look like there is a lot of room for other bits as well. I only saw a 1U model but do these guys have the same gear for larger more beefy servers? How about blades?
lastly how much does one of these things weigh?
"I am a kernel in the linux army"
The #1 on the top 500 supercomputer list is using water cooling as well (in combination with phase change cooling). Watercooling whole racks can be done. The only difference from TFA is that is also adds immersion cooling. Immersion cooling has been found to be superior in cooling but comes with (obvious) considerable maintenance problems. The video for this machine shows more or less standard water cooling blocks on the processors, along with various plumbing that to keeps the machine chilled.
The technique of using cheaper off-peak energy to freeze liquid and then use that liquid for daytime cooling loads is already used in a very few places. Combine that technique with the direct server cooling mentioned in the article and....wait a minute....they are already claiming a 93% cooling cost cut? Either their is huge waste now or they're already expecting to use off-peak energy. But then again, maybe the remaining 7% is still large enough to merit further savings.
Direct cooling makes far more sense than cooling rooms like I keep seeing around now.
LoB
"Anyone who stands out in the middle of a road looks like roadkill to me." --Linus
There are probably great economies of scale for datacenters, but what about Joe User? The article wasn't clear if 'included in the manufacturing process' would include consumer level systems. Just thinking that cost savings for datacenters is great, but I'd be really interested if it helped out the regular consumer (not to mention what kind of operational issues might this bring up?).
mu
There's a joke somewhere about your server being so ugly you have to put a bag over it before you go inside, but I can't quite work it. Help?
Yet another way to increase the density of server farms... Useful if you must grow your servers in Manhattan, a waste of money otherwise.
Among the many great things the internet has brought us (*cough*porn*cough*), "location-independence" ranks pretty high up there. Your servers don't need to all fit in one cargo container that runs so hot it requires LN cooling. For all it matters, you could put them in a single line of half-racks on a mountain ridge, cooled naturally by the wind (with some care to keep them rain-free, of course).
I thought we'd learned our lesson in that regard when tests last year by MS and Intel (not to mention Google's truly inspiring data center designs) showed a substantial payoff by letting servers run hotter and less densely packed. Silly me.
Source for excerpt below
"Intel set up a proof-of-concept using 900 production servers in a 1,000 square foot trailer in New Mexico, which it divided into two equal sections using low-cost direct-expansion (DX) air conditioning equipment. Recirculated air was used to cool servers in one half of the facility, while the other used air-side economization, expelling all hot waste air outside the data center, and drawing in exterior air to cool the servers. It ran the experiment over a 10-month period, from October 2007 to August 2008.
The temperature of the outside air ranged between 64 and 92 degrees, and Intel made no attempt to control humidity, and applied only minimal filtering for particulates, using "a standard household air filter that removed only large particles from the incoming air but permitted fine dust to pass through." As a result, humidity in the data center ranged from 4 percent to more than 90 percent, and the servers became covered with a fine layer of dust.
Despite the dust and variation in humidity and temperature, the failure rate in the test area using air-side economizers was 4.46 percent, not much different from the 3.83 percent failure rate in Intel's main data center at the site over the same period. Interestingly, the trailer compartment with recirculated DX cooling had the lowest failure rate at just 2.45 percent, even lower than Intel's main data center."
And although the failure rate was similar, the electricity bills were night and day. So I'm not buying into this unless your running a HUGE data warehousing op with more transactions than WalMart...
The problem with this is that it requires server manufacturers to standardize their designs. There was talk a few years ago about standardizing Bladeservers. I don't see this happening as there's too much control in the bladecenter chassis, switch interfaces, management abilities etc. Plus why would IBM want to sell an empty chassis and then let the customer fill it with HP C-Class blades?
Even racks themselves from IBM/HP/Dell/EMC/netapp/Sun aren't standardized, other than they are 19" wide. This is why if you mix vendors in the same rack you've got to adjust the depth of the rails.
As for going out and buying third party cabinets (APC for example), some of these have complex ductwork associated with them which makes them take up more than one tile of width.
These guys probably want two things, either IBM/HP/DELL license their technology or someone buys the company. Also, last I checked, there's not a large amount of room in my servers.
"Earlier this year, IBM predicted that in ten years all data centre servers might be water-cooled."
The costs of cooling air will be replaced by the costs of obtaining water. This system will not be for "water challenged areas".....Californy, etc.
In post Patriot Act America, the library books scan you.
In all honesty, this being a cool concept and all, it would not work in the real world because a) it cannot be retrofitted to existing systems and b) it requires the use of proprietory, unknown hardware. How many large companies are going to switch from tried and trusted server providers (like HP, IBM, Dell and as of late Cisco) in favor of something that, well, looks nifty. Their only shot at this not becoming vaporware is to try and sell the technology to a major server manufacturer, and even then I doubt it will work - imagine all the effort it would require to retrofit your existing data center for liquid cooling... liquids and server rooms don't go well together.
Bow before me, for I am root.
What could possibly go wrong?
Ok, so they are British and they spell 'center' with the 'er' the other way around. Why don't they spell server as 'servre'?
...but I've seen water cooling for my short time on this earth as the superior cooling method. So much that I ran it myself. Sony even puts it in their system once but it was of a passive system than a pump/coolant system. Big name using it right out of the box. I don't know why leaks would be that big of an issue as this isn't a high pressure water system; being a closed loop it is going to be a very low pressure system unless you are trying to blow water as fast as you can through it. If it moves too fast, it will create a layer of stagnant coolant just off the surfaces and degrade cooling. Low (pressure)and slow (moving) should yield best cooling. No need to move 2000 Lph unless you are using one pump for many heat sources to maintain flow, but I wouldn't put that many devices on one pump.
Wow, amazing, they finally produced something like what has been done on my website more than 9 years ago:
http://www.octools.com/index.cgi?caller=articles/submersion/submersion.html
I gave up with the idea of an useful sig...
I remember back in 1974, the IBM mainframe was water cooled. Worked pretty well until on Saturday the primary admin came in and forgot to turn on the water supply before starting the system. :)
I'm surprised that hot chips don't already include a layer of microfluidics right inside the package. People have been dealing with overheating chips inefficiently for years. There's clearly an opportunity to sell chips with fluid cooling built right into them.
I think eventually buildings will have fluid cooling systems attached to heat sinks for all kinds of purposes. Geothermal heat pumps already are popular for making heating and cooling up to 4x as powerful as the electricity powering them (instead of typical efficiency under 100%). Refrigerators and clothes dryers could also benefit in efficiency by routing their relocated heat into a fluid through a heat sink. Computers, TVs, and other electronic devices all could run more efficiently connected to a shared heat sink circuit, while avoiding heating air during hot seasons that uses energy to be cooled back down.
Ultimately, we'll have to find a way to consume the waste heat instead of just move it "away", or suffer the fate of Larry Niven's Puppeteer Homeworld. But in the meantime, we can do a lot better job managing it with the tech we've already got, with a few tweaks and more widespread application.
--
make install -not war
how about just cutting down the ac to dc to ac to dc part and make a common DC bus with the big and hot ac to dc part away from the severs and they can just have not as well dc to dc in them.
water has a lot that can go bad with it and do you want some water to mess up a $1000+ sever?
I had a start up, Nisvara Inc. 2002 - 2006 We had water cooled and could run whole server rooms with no air conditioning at all! Even had a partnership with NASA Ames.
Our system used sealed copper tube, and something I called a thermal ground, basically a copper or aluminum plate with the tube bonded too it. Then shims that connect the heat sources, the CPU, Northbridge, Southbridge and CPU Power supply and possible ram. Powersupply and hard drives were also connected to the plate to remove the heat.
We had many meeting with all the big players, Intel, Siemens, Sun, Maxtor, Pac Bell to name a few. None would allow water cooling in data centers. The liability for damaged equipment is too high.
We did come up with a lower cost fluorinert like solution that we could use, but still getting them to eliminate air conditioning was a very hard sell at the time. Also to including the extra plumbing and what not.
Maybe today they might start to change there attitude but I am not so sure about it.
http://web.archive.org/web/20040901070743/http://www.nisvara.com/
I am always doing that which I can not do, in order that I may learn how to do it. - Pablo Picasso
I had my own water cooling experiment about ten years ago. I had a two processor Athlon board and made two aluminium waterblocks for it. Since my metalworking skill was pretty low (and I was limited to hand tools), the blocks leaked, necessitating several patches. First with duct tape (:-), then with plumber's caulk, and finally by covering the whole thing with fiberglass epoxy, which plugged it up. Up to that time I had a nice little waterfall going from the waterblocks down onto the graphics card (a Radeon), onto the network card below that, and finally pooling at the bottom of the case. Surprizingly, the computer kept on working just fine for years, in spite of being constantly drenched. Then I got sick of messing with the plumbing and installed a fan, but then the motherboard failed after only a few months. Go figure.
No spill (as in "almost insignificant", not as in "not too much, won't empty the whole system, but you better have some towel nearby just in case"), quick disconnect, low resistance valves for watercooled system have been already available for quite some time for enthusiasts.
(Koolance is an example of compagny producing such thing in the US, Aquatuning is an example of shop selling similar implements in the EU - no links to avoid gratuitous advertising to web spiders, but you can easily google the names).
Anyway, low conductance liquids are popular in application where spills and leaks aren't easily monitored (see above source). And don't forget that every other blade module is sealed too. So in case of leak you're just spilling... on a sealed container which isn't affected by external liquids anyway.
As for pressure : Well, uh, no. You would need tremendous pressure if you had to fill the whole rack using 1 single pump. Which would be a single point of failure and is bad.
The more sensible approach would be each blade module having its own small pump (Laing DDC for the win !!!) for pumping water out of the rack's main tank.
It's already the scenario used in most rack-cooling situation (see again mentioned sources above). And in case of pump failure, well, only 1 blade module fails. The rest of the rack is immune to it.
Well I'm sure most /.er have some ricer friend (the kind which custom hand compile gentoo with "-O9999" :-) ) to whom a massive failure of watercooling has happened some time ago. Watercooling safety has evolved since then and it's now much more secure even for simple enthusiast. Now, a company specializing into data-centers has even more possibility to offer safety.
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
Why not cool data centers remotely, via wi-fi? The heat could be transferred wirelessly to another location far far away.
Either their is huge waste now or they're already expecting to use off-peak energy.
The current situation IS a huge waste.
Basically we're currently trying to cool down server using a fluid which is a *thermal insulator* (air). Of course it's catastrophically inefficient. But it happens to be simpler.
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
Going DC doesn't save as much as you might think
There was an apc paper (search for DC on http://www.apc.com/prod_docs/results.cfm?DocType=White%20Paper&Query_Type=10 to find it) on this not long ago.
They considered five systems, three existing and two hypothetical and looked at the total efficiancy including UPS, distribution and PSU in equipment (remember even with DC distribution you still need a PSU and generally said PSU needs to be isolating).
* american AC: 480V/277V three phase from the UPS converted to 208V/120V three phase by transformers in the PDU and then split out to 120V single phase to feed into the PSU. Overall efficiancy 83.56%
* european AC: 400/230 three phase from the UPS split out into 230V single phase to feed into the PSU. Overall efficiancy 86.39%
* telco DC: 48V DC from UPS straight to the PSU. overall efficiancy 84.58%
* hypothetical 380V DC: 380V DC from UPS straight to the PSU 87.64%
* hypothetical hybrid DC: 575V DC from UPS to an intermediate converter which converts it to 48V to feed to the PSUs 80.74%
The hypothetical 380V DC system wins but not by much and it is likely to be FAR FAR more expensive to deploy than the european AC system.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
I looked at specs for a new data center last week and the cost of electricity for the servers is followed closely by the cost for electricity to run the HVAC equipment. In a few more years it is likely the will become HVAC the major cost. So from a cost point of view the "lock in" is the HVAC equipment will become the major problem. This type of system will start to look real attractive and if we can get good leak detection within the server cabinet most of the problems will be manageable.