Slashdot Mirror


Cirocco Live Liquid Cooled Rack

Mark Grant writes "Cirocco have developed a liquid cooled rack of AMD Duron 1.1Gs in a Beowulf cluster. The rack has been installed in Cambridge University, England and has been under trial since Christmas. The system is being put through its paces running chemical research algorithms. Critical to Cirocco's liquid cooling system are the hot swappable quick couplings. These allow servers to be disconnected whilst the cooling system is in operation." The graph with live temperature readings is pretty neat.

63 of 119 comments (clear)

  1. Just Imagine by Anonymous Coward · · Score: 5, Funny

    Taking just one of those from the cluster.

    it would be like have a pc.

    wouldn't it

  2. Hot swappable? by questamor · · Score: 5, Funny

    The idea of "Hot swappable" when it comes to cooling couplings is making my head spin.

  3. Hmmmmm by GeorgeH · · Score: 3, Funny

    You have to wonder if "hot swappable" is the right term for this kind of system.

    --
    Why can't I moderate something "Wrong" or at least "Grossly Misinformed"?
    1. Re:Hmmmmm by RalphBNumbers · · Score: 1

      Since you'd be removing part of you're cooling system, I'd say " hot swapping" is an even more appropriate term than ever.

      --
      "The worst tyrannies were the ones where a governance required its own logic on every embedded node." - Vernor Vinge
  4. Quite Dangerous by Anonymous Coward · · Score: 3, Interesting

    While it's possible to secure this sort of thing so that water doesn't leak out from the cooling system, it's much harder to manage a _very_ large cluster of processors cooled by water. Cooling a cluster with air is easier, because you only have to watch to see if a fan breaks down, and that can be automated. Cooling a cluster with water is more dangerous in that you really should inspect it visually every so often to see if moisture is leaking out from the couplings (or condensing on the pipes). So really, could you imagine having to maintain a Beowulf cluster of these?

    1. Re:Quite Dangerous by Blaine+Hilton · · Score: 1

      I completly aggree. Liquid is frowned upon inside server rooms, I could not see pumping gallons of coolent through millions of dollars worth of equipment. Also you can easily add more air movement, but upgrading a pump system is much harder and requires a complete overhual. N+1 redudancy would be just about impossible also.

    2. Re:Quite Dangerous by tenman · · Score: 1

      from the website...

      Each cpu dissipates just under 50W which is traditionally air cooled using a large heatsink and fan. This is fine for a stand alone computer but when multiple computers are used, eg a Beowulf cluster, the rise in room temperature and hotspots are an increasing problem. Normally air-conditioning is used but this is very inefficient.
      Cirocco directly cool the heatsink with water which can be cooled remotely and recirculated.

    3. Re:Quite Dangerous by tenman · · Score: 1, Informative
    4. Re:Quite Dangerous by Blaine+Hilton · · Score: 1

      Yes I know, but there are still many questions. What happens if the dry switch fails?

    5. Re:Quite Dangerous by John+Zebedee · · Score: 1

      Presumably it would take a BC of plumbers?

      --
      The future is here. It's just not evenly distributed yet. -- William Gibson
  5. nice rack! by joe_bruin · · Score: 4, Insightful

    didn't cray master the liquid cooled cabinet design, like, 30 years ago?

    1. Re:nice rack! by scottm52 · · Score: 1

      Kind of.....

      Cray used (uses?) a liquid freon system where each board had a mettalic layer that connected into a slot formed with the cooling tubes. Thus "sinking" the unit to the coolant frame itself.

      This system (the Cray one) did not "re-plumb" liquid in/out of cases.

      So (IMHO) Cray "mastered" the way to do it. This is just another way to attempt the same effect with different hardware.

    2. Re:nice rack! by Curl+E · · Score: 2, Informative

      The Cray T3E is cooled with fluorinert. The heat is then dumped into cooling water with an external heat exchanger unit. The processor element modules (PEMs) - a board with 8 alpha processors 4 on the bottom mounted against an solid aluminium block and 4 on the top mounted upside down to the same block - slide into the processor cabinet and have quick release cooling hose couplings.

      --
      Backups are for wimps. Real men post their data in comments and have slashdot mirror it
    3. Re:nice rack! by Mark+Grant · · Score: 1

      Yes, Cray, IBM, HP have all used liquid cooling in the past, normally with exotic fluids (get out the chemistry notes). The major difference is that most previous systems were plumbed up like a house. Servers in racks need to be hot swappable or physically removable from the system.

  6. Liquid cooled rack? by sssmashy · · Score: 5, Funny

    Personally, I always thought a liquid cooled rack is what happens when Pamela Anderson spills beer down her shirt...

  7. Fluid connectors by Gordonjcp · · Score: 3, Insightful

    Do those special, magic, fluid connectors look like scaled down versions of ordinary hydraulic dry-disconnect spools to anyone else?

  8. Re:Wow! by Old+Wolf · · Score: 2, Interesting

    serious question - is it possible to have a Beowulf cluster of Beowulf clusters?

  9. Chemical research? by moosesocks · · Score: 4, Funny

    I wonder what type of chemical research these systems will be conducting... perhaps they will determining the reaction between water (H2O) and Silicon printed circuit boards? (come to think of it, reasearch isn't the only thing they'll be conducting)

    --
    -- If you try to fail and succeed, which have you done? - Uli's moose
    1. Re:Chemical research? by JesseL · · Score: 1

      Printed circuit boards are (usually) made from fiberglass, resin, copper, and tin. Silicon goes in the integrated circuits soldered to the PCB.

      --
      "Prefiero morir de pie que vivir siempre arrodillado!"
  10. maybe... by deadsaijinx* · · Score: 5, Interesting

    I'm missing something. As cool as it is, why do you need to liquid cool 1.1Ghz Athlons. Its nothing a fan can't handle adequitely and at a much more desireable cost. Are they just going for the wow factor, or is there an actually reason for the liquid cooling.

    Was going to make a beowulf joke, but then you insensitive clods would mark me redundant (I'm only like the 20th poster, how redundant can I be?)

    --
    YOU SUCK BALLS!
    1. Re:maybe... by aliens · · Score: 1

      I was wondering the same thing. Not only that, but they're just Durons... Unless they were donated I would much rather have spent the little extra and gotten actual Athlons. Even if it meant fewer. Unless they needed a certain number of processors for some reason.

      --
      -- taking over the world, we are.
    2. Re:maybe... by benwb · · Score: 2

      You should be able to get a much denser rack without having to worry about proper air flow.

    3. Re:maybe... by neonstz · · Score: 1

      I'm missing something. As cool as it is, why do you need to liquid cool 1.1Ghz Athlons. Its nothing a fan can't handle adequitely and at a much more desireable cost. Are they just going for the wow factor, or is there an actually reason for the liquid cooling.

      I'm no expert at this (and I didn't read the entire article), but it by using liquid cooling instead of fans you can stack a lot more CPUs into the same space. Getting rid of the heat would also be easier since you can put the radiator somewhere, like outside or in another room.

    4. Re:maybe... by questionlp · · Score: 1

      Although their site seems to be down, but Angstrom has a 1U server that supports quad Athlon processors in a single case by having two Athlon MP motherboards... mind you, all in one rack unit. That's definitely a feat, even without liquid cooling, but that machine must be one loud, whiny and heat soaked machine... mostly if you want to stick 40-42 of those in one cabinet.

    5. Re:maybe... by ComputarMastar · · Score: 5, Insightful
      From the site:
      Each cpu dissipates just under 50W which is traditionally air cooled using a large heatsink and fan. This is fine for a stand alone computer but when multiple computers are used, eg a Beowulf cluster, the rise in room temperature and hotspots are an increasing problem. Normally air-conditioning is used but this is very inefficient. Cirocco directly cool the heatsink with water which can be cooled remotely and recirculated.
      Air cooling works well enough until you get many hot devices in a small space. Then you have the problem of some running too hot because the air thats supposed to be cooling them is already hot from cooling others.
    6. Re:maybe... by moosesocks · · Score: 1

      I have a duron 1.1 system. You'd have to stack a hell of a lot of these on top of each other to require any sort of special cooling.

      I have a small $5 aluminum cooler-master heatsink with a underpowered fan on it which is able to keep the CPU comfortably cool (without a case fan)

      I suppose it could get a bit tight in a 1u configuration, but anything more dense would probably require special hardware.

      Speaking of special hardware, why can't they just squeeze a bunch of mini-itx motherboards into a server case? They could use IDE compactflash drives to cut down on cost, heat, and space, as beowulf nodes only need enough storage to boot the kernel. I suppose you could fit 4-6+ of these into a 1u case, but you'd probably have to leave 1/2u space between cases to allow for ventilation (bringing it down to a 4x increase in density)

      --
      -- If you try to fail and succeed, which have you done? - Uli's moose
    7. Re:maybe... by virtual_mps · · Score: 1

      Ok. Your fan can cool 1 Athlon, no problem. The question is how you'd cool, say, a fairly densly packaged system of 1500 Athlons in 10 racks. (8 procs per 2u.) That's 75kW. Liquid cooling is simply more practical once you start packing enough hot processors into a small space. (This sort of density isn't implausible, it's only a short evolution from what cray was achieving in the T3E 8 years ago--and they did it with liquid cooling.)

    8. Re:maybe... by AlecC · · Score: 1

      In order to air cool, you need quite a considerable volume of airspace above the CPU and probably a head sink as well, and a relativewly unobstructed airflow from front to back of the PCB. (Some opeople do side to side, but usually not more than once). This is fine for one or two CPUs, but it makes the CPU and its cooling space effectively about two inches tall. Also, you cannot have one CPU pehind another in the airflow direction, because the second will have its air preheated by the first. For supercomputing use, the processor only required ram and network IO. This means that each kernel can probably be squeezed into an area about 3 inches square - which means that, cooling aside, you coult pack them three or four deep between front and back of the back. The graphic looks like a blade architecture. With air cooling, I would have thought you would get 1 CPU per blade on a 2 in spacing, or about 8CPUs per 2U rack unit. With warer cooling, you might get CPUs per blade on a 3/4 in spacing, or about 60 CPUs per 2U rack unit (hmmm - I think that is a bit optimistic...).

      Then, if you use air cooling, where does your heat come out? Into the server room. And if you have 256 CPUs at 50W each, that is over 10kW of head being dumped into your server room. You need some *serious* aircon to get that out of the room. And what is the aircon doing? Extracting the heat from the air and putting it back into some kind of fluid, which is piped to heat exchangers. So it makes sense to get the heat into fluid earlier rather than later.

      The way high-performance processing is going nowadays, I can see fluid cooling making a modest return to favour at the high end of the scale. It is already returning at the low end, with high performace laptops using fluid cooling instead of fan cooling. So I don't think this is a distracion at all.

      --
      Consciousness is an illusion caused by an excess of self consciousness.
  11. Mod -5 Don't Let The Other Geeks Know I'm Clueless by nick_davison · · Score: 4, Informative

    For those of use who've grown used to making "Woo, imagine a Beowulf cluster of them!" jokes yet have no clue what a Beowulf cluster actually is, the definition, history and so on is available at:

    NASA's Beowulf site

    In brief overview:
    In the summer of 1994 Thomas Sterling and Don Becker, working at CESDIS under the sponsorship of the ESS project, built a cluster computer consisting of 16 DX4 processors connected by channel bonded Ethernet. They called their machine Beowulf. The machine was an instant success and their idea of providing COTS (Commodity off the shelf) base systems to satisfy specific computational requirements quickly spread through NASA and into the academic and research communities. The development effort for this first machine quickly grew into a what we now call the Beowulf Project. Some of the major accomplishment of the Beowulf Project will be chronicled below, but a non-technical measure of success is the observation that researcher[s(sp)] within the High Performance Computer community are now referring to such machines as "Beowulf Class Cluster Computers." That is, Beowulf clusters are now recognized as genre within the HPC community./i

  12. Slashdotted... by MoTec · · Score: 5, Funny

    Perhaps they should look into using one of these for their webserver.

  13. Temp readings by Anonymous Coward · · Score: 2, Funny
    The graph with live temperature readings is pretty neat.

    Even more impressive when slashdotted.

  14. Yeah I Remember Those by ihatewinXP · · Score: 3, Funny

    yeah I used to have a Scirocco, water cooled and everything. A great Volkswagen but damn what a parts hog. 2.8 Liter engine and a 5 speed manual it was a blast to drive.

    Oh Cirocco? Not Scirocco? Whoops, but why on earth would anyone would dig up that dead convoluted name is beyond me. A few VW enthusiasts might always remember you but I think your just alienating your audience, all naming your company after trade winds.... Maybe iCirocco? Nah.

    --
    ---- The real Slashdot is still here. You just have to browse at -1 to read the comments.
  15. Re:Wow! by k-0s · · Score: 1

    Anyone think the editors were baiting us with this story. Before I even clicked "Read More" I knew just KNEW there was going to be a few "imagine a cluster..." posts. And if there wasn't I was going to make one, LOL.

  16. 1.1ghz chips need liquid cooling?!?!? by sakusha · · Score: 3, Insightful

    This is a stupid idea. They say that air conditioning is inefficient, but they could have easily done it efficiently with ductwork.
    I've worked with quick-couplings on megawatt lasers, and I can just give em one tip: couplings fail more often than computers. Just wait til they spring a leak because some idiot forgets to twist the ring properly, and he floods the whole rack.

    1. Re:1.1ghz chips need liquid cooling?!?!? by stratjakt · · Score: 1, Insightful

      If the design was good, the couplings would be located such that if one leaked, it wouldnt possibly get anywhere near the actual electronics. Like outside and below the case.

      --
      I don't need no instructions to know how to rock!!!!
  17. Don't say it. by SandSpider · · Score: 5, Funny

    You know what I'm talking about. Don't do it.

    --
    There is nothing so good that someone, somewhere, will not hate it.
    1. Re:Don't say it. by spanky1 · · Score: 1

      Too late... about 8 people already posted the standard "imagine a beowolf cluster of these"...

    2. Re:Don't say it. by SandSpider · · Score: 1

      I'm hoping to stop the next 50, though. But I'm an optimist.

      --
      There is nothing so good that someone, somewhere, will not hate it.
  18. Re:Mod -5 Don't Let The Other Geeks Know I'm Cluel by Shadow+Wrought · · Score: 4, Funny

    You know, in a way I'm disappointed. I had always hoped to find out that Beowulf Clusters only acheived prominence after having slayed the evil Grendel Clusters in the back of some Scandanavian CompSci lab...

    --
    If brevity is the soul of wit, then how does one explain Twitter?
  19. Nice rack! by Junior+J.+Junior+III · · Score: 1

    I wouldn't mind having one of those to get my juices flowing, if you know what I mean. Wink Wink, Nudge Nudge.

    --
    You see? You see? Your stupid minds! Stupid! Stupid!
  20. Re:Mod -5 Don't Let The Other Geeks Know I'm Cluel by nick_davison · · Score: 3, Funny

    Wow! Thanks for providing a link that's in the article. You're the best.

    <tongue firmly in cheek>
    I'll be gentle, seeing as you're clearly a newbie...

    In time, when you've had a chance to read more heated arguments on Slashdot, you'll soon learn that we rarely take the time to actually read the articles. We just get on with ranting on them. I blame Hillary Rosen, the RIAA and Microsoft for that - in no particular order.

    Now, if you'll just go to the second doorway down the hall, I hear there's an opening for a Grammar Nazi.
    </tongue firmly in cheek>

  21. Re:Liquid cooled rack? by outsider007 · · Score: 1

    actually if you substitute natalie portman for pam anderson and hot grits for beer then this post has it all!

    oh yeah, in soviet russia all of us are belong to your base.

    --
    If you mod me down the terrorists will have won
  22. Did anyone else think... by Dave21212 · · Score: 3, Funny


    ...that "Live Liquid Cooled Rack" was some sort of wet T-shirt contest for geeks ?

    Seriously though, match this with the IBM Ice-Cube storage cluster and you really would have one cool machine (ducking).

    "In a few years, one storage administrator should be able to manage a petabyte of storage, which is 100 times more than is typical today." - IBM Almaden Research Center

    --
    "Whoever would overthrow the liberty of a nation must begin by subduing the freeness of speech."--Benjamin Franklin
  23. Re:Jobs in computational chemistry by SatanicPuppy · · Score: 1

    Organic chem is good. Don't know about the rest. Make sure you know what "genomics" means.

    --
    ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
  24. Why...? by cr0sh · · Score: 1
    I understand that mounting multiple machines in this fashion would necessitate a different cooling solution, but I still cannot understand why liquid cooling would be the proper one? I would think a properly designed air-flow cabinet would be the cheapest and most easily maintainable solution.

    I am thinking something like a front-to-back fan design to blow the hot air out of the 1U cases (and I know those cases are pretty tight and cramped, so intelligent routing of cables, or custom device interconnects would be in order too) to the rear of the rack, and into an integrated (with the rear door) duct that funnels the hot air (with a fan) into the ceiling (or outside). Perhaps a better solution would involve heat pipes leading away from the CPUs to the duct (less fans).

    I guess what I am trying to get at is that an intelligent design on the part of the 1U cases and the rack could probably acheive great results with air-cooling alone, and be cheaper for the end-user of the machine (from both an initial cost standpoint, as well as maintenance) as well.

    I am not saying that liquid cooling doesn't have it's place, but this doesn't seem it (of course, I could be completely wrong, as I have never worked with such a dense system before - I do know server rooms can heat up quick, even when they aren't filled - so a dense rack very well might be a big issue and need liquid cooling)...

    --
    Reason is the Path to God - Anon
  25. Slashdotted? by Oculus+Habent · · Score: 1

    Hey, The temperature is increasing on the the cluster, probably as more people are checking out the live temperature monitor. Also, it seems to be loading slower.

    You can literally watch the slashdot effect on a server this way.

    --
    That what was all this school was for... to teach us how to solve our own problems. -- janeowit
  26. cooling by poot_rootbeer · · Score: 1


    Why not just install rackmounting rails on the inside of a refrigerator? That's what I would do if I were stupid.

  27. Plwase no beowulf cluster jokes. by Unregistered · · Score: 1

    It already is one, a beowulf cluseer would be just the same but bigger. Move along.

  28. Why use water by Smallpond · · Score: 3, Funny


    Even though the heat capacity of gases is generally larger than liquids, the thermal conductivity of water is about 30 times larger than air. Also, plumbing lets me move the water to exactly where I want the cooling to take place without heating it along the way.

    IBM mainframes (ECL-based) used water-cooled plates for the CPU and have spent a lot of design effort on quick-connect couplings that don't leak. I just wish they had transferred some of that knowledge to the Sears washing machine group.

  29. Why 1.1GHz Durons? by gilesjuk · · Score: 1

    The later Athlon XP cores (thoroughbred) run much cooler at higher clock speeds and faster. So downclocking such a chip would result in less cooling needed.

  30. Woah... by KevinIsOwn · · Score: 1

    Imagine a Beowulf cluster of... damnit.

  31. How big a cluster? by hopbine · · Score: 1

    I just took a quick look at the site (at 7:09pm est one of the machines, clinux7, showed at 55C - wonder what happened) and it only shows 9 machines. Apart from that its a very neat idea if you consider the efficiency of water vs air cooling. Why chill a large room when all that you need to cool is a small chip. The trouble is one chiller and many small areas, solution is to have some way of disconecting an area (machine)if you have to do some kind of repair. I think that I would have use taps, and out all them all in parallel, but I'm not a plumber.

    --
    Semper ubi sub ubi
  32. Re:Mod -5 Don't Let The Other Geeks Know I'm Cluel by NedTheNerd · · Score: 1
    so what your saying is a beowulf cluster is like a bunch of computers? I allways though it was in close relation to the wolf having pink hair and only seen one day of the year. in a small town of mexicans that have no paticular contact with the outside world. so i was wrong?

    *runs from the flames*

  33. Re:Liquid cooled rack? by JDWTopGuy · · Score: 1

    In soviet russia, your sig says "Super lemons clone YOU!" And aren't the grits supposed to go in the pants? I don't think you can compare Portman's rack to Anderson's.

    But seriously, go easy on these jokes. I found out the hard way that nobody but you and me find them funny anymore. My karma's been like a yo-yo lately. (here it goes again!)

    P.S. You forgot the beow... never mind.

    --
    Ron Paul 2012
  34. 50 watts? by NedTheNerd · · Score: 1
    50 wats isnt much at all (yea ya i know been said) but I was just thinking. With my current setup my cpu outputs around 172watts and with 120CFM fan on a heater core (thats the thing in your car that delivers heat) I get around 40~45C with an ambient of 25. do they use one enormous radiator or a few little ones cuz their temsp are pathetic (highest 43C) not to mention that all of those have peeked to 55C - 80C within the last HOUR! (supprised some of those are still working)

    geuse they know not to put gellow in it next time

  35. Re:Mod -5 Don't Let The Other Geeks Know I'm Cluel by deadsaijinx* · · Score: 1

    has someone been reading classic literature again? And then confusing sed literature with their own fantasies?

    I think the answer is yes

    --
    YOU SUCK BALLS!
  36. Liquid cooling is cool.... by resident-crank · · Score: 2, Informative

    many good reasons to use liquid cooling. firstly, it's *very* efficient and it allows very high volumetric density. secondly, there are times when air cooling is a Truly Bad Idea, like on boats, ships, and submarines, but also even "ground-based" transportation applications. It's pretty clear that blowing salt air over a circuit board (even with conformal coating) is a Bad Idea(tm), but it's also true in cars, trucks (aka "lorries"), and things like earthmovers. and in large systems, the efficiency part is a Really Big Deal. Heating up a huge flow of air on its way to the ceiling, just so the air chiller can try to move it back down to get sucked in again is just stupid except at very small scale. the Beowulf cluster in question may not *demand* liquid cooling, but you don't have to build one a lot larger for the difference to matter a lot. (This is especially true in Europe where rooms are not airconditioned to the degree they are in the US in the first place.) rather than upgrade an entire AC system, which probably involves lots of work on a 500 year old building, just run the loop to a remote chiller and declare victory. as for "it will leak", note that almost everything electronic in a submarine is liquid-cooled. it is true that the primary loop through the cold plate may not be water, but it gets to a water loop pretty quickly. no-leak connectors have been around a very long time - just not cheap ones. a large demand for better cooling technology is important to drive down the costs and to make it commonplace and not just the province of "Big Iron" (supercomputers or otherwise). -mo

  37. Mainframe cooling, again by Animats · · Score: 1
    In the IBM mainframe world, everybody, including IBM, was glad to get rid of liquid cooling. By the early 1990s, all IBM mainframes were air-cooled. This seems a step backwards.

    If cooling is a real problem, the usual solution in dense avionics racks is engineered airflow. All heat sources are measured, and small ducts and diverters are sized and built to deliver air to the key spots, while not wasting it on stuff that doesn't need it. If you're building rackmount servers as a product, it's worth the trouble to do this. Otherwise, most of your airflow is wasted.

  38. What cooling? by lptp · · Score: 1

    CPU's: 37-44, Ambient: 24, water in: 26.5, water out: 26...

    Isn't "cooling" supposed to create more than an accidental 0.5 degree temperature difference?

    --
    Caveat Emptor: this message won't selfdestruct if you memorize it!
  39. Duron vs. Athlon by ion++ · · Score: 1

    I would expect that they had tested Duron vs. Athlon in advance, and bought the cpu type and number that gave them the most bang for the buck.

    Suppose they only run stuff that can fit in the cache size of the Duron, then the extra price for the athlon is a bad idea.

  40. Durons? by Giggles+Of+Doom · · Score: 1

    I don't own A Duron, let alone many Durons that would necessitate an entire rack.

    --
    "A coward dies a thousand deaths, the brave but one."
  41. Re:Mod -5 Don't Let The Other Geeks Know I'm Cluel by Shadow+Wrought · · Score: 1

    Busted!

    --
    If brevity is the soul of wit, then how does one explain Twitter?
  42. Re:Mod -5 Don't Let The Other Geeks Know I'm Cluel by gregger · · Score: 1

    Actually Grendel (the book) isn't classical at all is it?

    Shows you what I remember from Schweitzer's class.

    TTFN