Slashdot Mirror

← Back to Stories (view on slashdot.org)

The Data Center Density Debate: Generational Change Brings Higher Densities (datacenterfrontier.com)

Posted by Soulskill on from the try-not-to-burn-out-the-grid dept.

1sockchuck writes: Over the past decade, there have been repeated predictions of the imminent arrival of higher rack power densities. Yet extreme densities have remained focused in high performance computing. Now data center providers are beginning to adapt their designs for higher densities. One of these companies is Colovore, which is among a cliuster of companies adopting chilled-water cooling doors for their cabinets (LinkedIn is another). They say the move to higher densities is driven in part by a generational change in IT teams, as younger engineers are less worried about high-density strategies using water in the data center. "A lot of them grew up with PC gaming and water cooling right in their living room," said a Colovore executive.

45 comments

  1. Water is for cows by Anonymous Coward · · Score: 0

    Moo.

  2. How in the hell is this a DEBATE? by Anonymous Coward · · Score: 0

    Density = good. Mitigating heat to increase density with a simple, reliable system = good. Anyone against increased density in datacenters... might be too dense?

    1. Re:How in the hell is this a DEBATE? by mwvdlee · · Score: 3, Insightful

      How reliable is it, really? What happens if a pump breaks down in a high-density liquid cooled rack versus a breaking fan in a low-density air cooled rack?
      How likely are those breakages? At the every least they are less than 100% reliable; they WILL break at some point. It might be perfectly fine, but a single PC at home does not scale to an entire datacenter of racks.

      --
      Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
    2. Re:How in the hell is this a DEBATE? by dwywit · · Score: 1

      The last quote in the headline got me:

      "A lot of them grew up with PC gaming and water cooling right in their living room," said a Colovore executive.

      Face, meet palm. Although it's good to know I can cross Colovore off the list of datacentre candidates.

      --
      They sentenced me to twenty years of boredom
    3. Re:How in the hell is this a DEBATE? by Z00L00K · · Score: 2

      Each new generation of electronics has a higher density, so this isn't really news.

      During the 70's a computer room was huge with a lot of stuff physically. Today that computing power fits in your pocket.

      In addition to this we have more and more data stored electronically, so the increased density is somewhat counteracted by that.

      Another factor is that earlier there were only a few terminals locally, but today we have networks giving every household a connection where a large number of terminals can reside. Soon your toaster is on the internet as well. All that counteracts higher density in the data centers because they need to be able to manage more connections and a higher traffic volume every year.

      And even when you get a higher density you need to cool the electronics in a way that avoids hot spots in the electronic devices shortening their life span.

      --
      If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
    4. Re:How in the hell is this a DEBATE? by jabuzz · · Score: 3, Interesting

      Speaking from experience here running an HPC system with water cooled doors on the back of my racks for just shy of four years now. My personal view point is if you are not doing racks with water cooled doors then you need the sack unless you are using one of the in rack water cooling systems.

      There is no pump in the rack/door. You hook up to the cool water supply that is coming from the chillers, and they provide the "pump". You are no worse off than before with air handling units in the room which also don't have pumps in them.

      The doors we use have a number of large aka ~400mm diameter fans in the back that spin relatively slowly, and these are N+1 redundant. Besides that they are the sorts of fans than can spin for decades before going wrong.

      The other advantage of having the cooling in the doors is that you save floor space in the room as you don't have to so many air handling units in the room.

      The first point to note is you probably still need some air cooling in the room. Things like tape libraries don't come with water cooled doors for example. Though there is an argument that the tape libraries should go somewhere other than you data centre.

      The second problem is suppliers that want to sell you something prepackaged in a rack that is not water cooled. Trying to explain to them that you don't want their shitty none water cooled rack is in my experience is like taking to a brick wall. Another reason why you probably still need air handling units. That said our doors are currently providing net cooling to the room, that is inlet temperature at the front of the rack is higher than the outlet temperature at the back.

      The third problem is strip off to tee shirt and shorts if you open the rack door while the system is going full tilt, it can be sufficiently hot.

      Now go crawl back under your rock and stop spouting nonsense about things you clearly know nothing about.

    5. Re:How in the hell is this a DEBATE? by NicknameUnavailable · · Score: 1

      My first liquid cooled computer was also the first computer I built. I had to take it apart and fix leaks at least once a month - it was higher maintenance than a Japanese sports car.

    6. Re:How in the hell is this a DEBATE? by Anonymous Coward · · Score: 0

      It's not so obvious. Why is density good? Wouldn't it be cheaper to use a low-density datacenter, filled with cheap, easy-to-cool components?

      Also, some operators have found that there are cost and energy savings to be had be letting the datacenter run hot, at the expense of reliability, and making up for it with redundancy.

    7. Re:How in the hell is this a DEBATE? by Anonymous Coward · · Score: 1

      Out of curiosity, what sort of equipment are you running that requires the water cooling?

      The article alludes to 20-30kW per cabinet, but that's pretty standard for air cooling these days. Cold isle, hot isle and ducted bottom to top cabinet cooling are the experience I have. We run cabinets of 4 x blade chassis with around 7kW per chassis at peak use.

      The article also mentions dreams of 250kW per cabinet, but doesn't suggest anyone is doing that yet.

      We run the rooms at a higher than usual ambient temperature, so it's tee-short and shorts all the time.

      We do have a cabinet (ducted) with a different form factor of blades which gets pretty toasty when the back door is open (can't tolerate it for longer than a few mins before needing a break). It runs just fine when the door is closed, though. These are lower density than blades, so they are just inefficient at getting their hot air out.

      The really fun part is when our cooling fails. The room temperature rises from 20C to over 40C in 10-15 mins. Rapid shutdowns get some good practice! Fun times. The same could happen for water cooling, though. Failures during maintenance being the most common.

    8. Re:How in the hell is this a DEBATE? by dbIII · · Score: 2

      My personal view point is if you are not doing racks with water cooled doors then you need the sack unless you are using one of the in rack water cooling systems

      So those of us that still have a lot of empty space available and don't have to change to high density yet "need the sack"?
      WTF is it with the insulting zealotry? It completely demolishes credibility, making any of your points questionable whether they are correct or not.

      Also you've said nothing useful about water treatment, condensation, descaling or corrosion. They used to be a pretty big deal back in the day and are today in other systems cooled with water, why not worth mentioning now?

    9. Re:How in the hell is this a DEBATE? by JaredOfEuropa · · Score: 1

      They may have a point, though I suspect it has more to do with the managers than with the engineers. I've seen plenty of managers sweep a proposed innovative change off the table because it goes against what they are used to: "Not on my watch!" The manager who has worked with networked PCs his whole life does not approve of thin clients and desktop streaming. Mr. Big Iron hates the idea of decentralized data processing. Then another manager comes along and approves the change. In this case, I can see a manager say "Water in the data center? Not on my watch", whereas the next guy who is already familiar with the concept (and may even have some hands-on experience) decides to go ahead with it.

      --
      If construction was anything like programming, an incorrectly fitted lock would bring down the entire building...
    10. Re:How in the hell is this a DEBATE? by JaredOfEuropa · · Score: 1

      Was this in the early days of water cooling? The technology has come along rather well and fittings and tubings have improved a lot (and have been more or less standardized as well). I built a water cooled gaming rig about 5 years ago because I wanted a quiet machine, and it hasn't had any major leak in years of intensive use (some coolant did slowly leak out and evaporate somewhere, as I had to top up the system every 6 months or so).

      Think about the plumbing in your house, or perhaps something closer to data center cooling: a house with hot water / radiator based heating. How common is it to have a major leak in that? Not common at all: some of the plumbing here goes back 30 years and none of it has leaked so far. Most leaks seem to come from people accidentally drilling into a pipe, frozen pipes bursting, or faulty washing machines. The rest of it is pretty reliable.

      --
      If construction was anything like programming, an incorrectly fitted lock would bring down the entire building...
    11. Re:How in the hell is this a DEBATE? by Anonymous Coward · · Score: 0

      That's probably because they don't really work in the industry they claim to represent. People working on those systems are usually intelligent. Check the way they write and it's a reasonable conclusion that they're full of shit.

    12. Re:How in the hell is this a DEBATE? by Anonymous Coward · · Score: 0

      The title is actually referring to generations of people, as in young new managers thinking differently than the older managers they are replacing.

    13. Re:How in the hell is this a DEBATE? by jabuzz · · Score: 1

      Well the other plus side is that you get a better PUE so regardless of whether you have plenty of floor space you are better off with a water cooled door on the back of the rack. Especially as you will likely have several generations of equipment go through the rack.

    14. Re:How in the hell is this a DEBATE? by Anonymous Coward · · Score: 0

      Datacenters are extremely expensive and locality and data locality is important.Why have a Hubble telescope when it's cheaper to have several Earth bound telescopes?

    15. Re:How in the hell is this a DEBATE? by jabuzz · · Score: 2

      Nothing requires water cooling, it is just a water cooled door on the back of the rack gives us a better PUE, we don't waste floor space on air handling units and we don't have to worry about getting all the air containment for hot/cold isles either.

      Off the top of my head we are about 35kW per rack we the nodes are all spun up, though of course 100% utilization is super hard because we are always having nodes idling while to gather enough free for the next job to run. Nodes are about 85-90% busy.

    16. Re:How in the hell is this a DEBATE? by mikael · · Score: 1

      Pictures are more helpful to explain. The piping to the water cooling:

      https://images.duckduckgo.com/...

      The cooling water pipes goes all around the back door of the rack. Water goes in one pipe and out the other. Then the heat is carried off using an air to water exchanger. What worries me is whether they are recycling the water or just wasting the mains water supply.

      --
      Vintage computer adverts: http://www.vintageadbrowser.com/computers-and-software-ads
    17. Re:How in the hell is this a DEBATE? by drinkypoo · · Score: 1

      My first liquid cooled computer was also the first computer I built. I had to take it apart and fix leaks at least once a month - it was higher maintenance than a Japanese sports car.

      You probably should have used automotive components, then. I've got coolant lines thirty years old and not leaking. (I've also had to replace coolant lines at ten years or so, but never worse than that.)

      If I were trying to build liquid-cooled computer cabinets, I'd go to automotive supply for my parts and chemicals. That stuff can commonly last decades without maintenance, and you can readily order replacements. But the truth is that I've just stuck with air cooling, and not regretted it. It's not like my computer has to sit in traffic.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
    18. Re:How in the hell is this a DEBATE? by Anonymous Coward · · Score: 0

      Um, we DO have several earthbound telescopes.

      A datacenter is just a boxlike warehouse-type building in the middle of nowhere, where land is cheap. All the cost is the stuff inside the datacenter.

      It doesn't really matter how densely you arrange the same shit. There's no real downside to just making the darn thing bigger.

    19. Re:How in the hell is this a DEBATE? by VVelox · · Score: 1

      > The second problem is suppliers that want to sell you something prepackaged in a rack that is not water cooled. Trying to explain to them that you don't want their shitty none water cooled rack is in my experience is like taking to a brick wall. Another reason why you probably still need air handling units. That said our doors are currently providing net cooling to the room, that is inlet temperature at the front of the rack is higher than the outlet temperature at the back.

      What bloody suppliers?

      Only a handful of companies sell products that include a actual rack and from my DC experience over the last decade, few companies buy those.

      If you mean buy a massive group of blade servers? HP, Dell, or Supermicro really don't care what sort of rack they are going into... although Dell and HP would prefer to sell you their brand of cabs...

    20. Re:How in the hell is this a DEBATE? by Anonymous Coward · · Score: 0

      Even in the early days, you had to do something *really* stupid to get

      leaks at least once a month

      And I'm talking "homebrew reservoir, milled copper CPU block, heater core rad, Eheim pump, PVC hose with simple barb fittings" early days.

    21. Re: How in the hell is this a DEBATE? by Anonymous Coward · · Score: 0

      Emc, Hitachi, IBM to name a few

    22. Re:How in the hell is this a DEBATE? by plopez · · Score: 1

      What happens if the chiller break breakdown? do you have a way to take the racks down gracefully if the chillers fail? And why water and not CFC based? Since CFCs are not conductive it would seem to be a safer choice. BTW are you using tap water or de-ionized water?

      --
      putting the 'B' in LGBTQ+
    23. Re: How in the hell is this a DEBATE? by Anonymous Coward · · Score: 0

      Experience isnthe reason why this is taking off - or rather, lack thereof.

      Yes, there are benefits and the technology to liquid cool continues to improve. However, there is still increased margin on catastrophic failure disaster, and increased cost and maintenance for the equipment. You still need all the old systems in triplicate, but you also need additional intermediate systems to interchange. And like magic smoke, you still need additional maintenance and precaution to make sure it remains contained.

    24. Re: How in the hell is this a DEBATE? by Anonymous Coward · · Score: 0

      I think what he was referring to was data locality. Web data served upnfrom Kansas to LA is slow.

    25. Re:How in the hell is this a DEBATE? by Anonymous Coward · · Score: 1

      Yeah, I bet a lot of them were used to eating food right next to those home computers and watching porn in them. Doesn't mean that those are acceptable business practices due to familiarity.

    26. Re:How in the hell is this a DEBATE? by dbIII · · Score: 1

      Humour me because the only water cooling I've had anything to do with was in power stations in the 1990s before I got into IT, but what do you have to do to ensure that the rack will survive long enough to have several generations of equipment go through it? Is it as simple as an additive or do you need demineralized water as well? Are there sacrificial anodes you need to replace after a few years?

    27. Re:How in the hell is this a DEBATE? by jabuzz · · Score: 1

      How often do you replace a radiator in your house? How often do you for that matter replace a radiator in your car? The life span of a single generation of IT equipment is say five years maximum, so even being conservative a rack should be good for at least 15 years or three generations.

    28. Re:How in the hell is this a DEBATE? by dbIII · · Score: 1
      So you are dumbing down industrial to domestic as well as saying we all need the sack? I put an additive in my radiator in my car. Do you need to do the same for this situation or do you not know either way so are avoiding the question?

      should be good for at least 15 years

      Based on what exactly?
      I suppose I shouldn't expect much when you told the earlier poster to "go crawl back under your rock" just for asking a few questions, but please could we have a little bit more than something along the lines of "because I said so" if you really do know a bit about this sort of cooling in practice while the rest of us do not in this situation.

    29. Re:How in the hell is this a DEBATE? by StikyPad · · Score: 1

      He doesn't care because he won't be there in 15 years when other people are cleaning up his mess.

      --
      https://www.eff.org/https-everywhere
    30. Re:How in the hell is this a DEBATE? by Anonymous Coward · · Score: 0

      Great picture, I can see about two dozen potential problem areas on that fitting alone. The question to me seems not if something will fail at some point, especially over multiple generations of equipment:but how catastrophic will the eventual failure be.

  3. If it works for diskc by Anonymous Coward · · Score: 0

    It can and must work as well for DCs. A Newtonian law of the first order if ever there was such a thing.

  4. lmao by Anonymous Coward · · Score: 1

    when did we start taking advice on how to run a data center from "gamers" with liquid cooling systems?

  5. Pfft by Anonymous Coward · · Score: 1

    The last international data center company I worked for had water based fire suppression.

    Saved them a ton of money, at least until one of the data centers caught fire. The whole DC was a loss. When the final fire investigation was finished and the determined cause was the wiring done by an non-licensed electrician (Another cost savings) the insurance refused to pay.

    So, put water cooling in the rack and stack that power density as high as you want. Just remember they are cutting every other corner to save a buck! I am sure the new L1/L2/ and L3 guys in India will take care of it.

  6. Clickbait... by msauve · · Score: 4, Informative

    Don't bother. The article is clickbait. The submitter has a history of frequently submitting articles from the same website. I suspect 1sockchuck is affiliated and not disclosing that fact.

    --
    "National Security is the chief cause of national insecurity." - Celine's First Law
    1. Re:Clickbait... by Zontar+The+Mindless · · Score: 1

      I don't see anything that looks like affiliate info in that link. And you can copy/paste the URL into a new browser tab so there's no referrer info, either.

      --
      Il n'y a pas de Planet B.
    2. Re:Clickbait... by Hadlock · · Score: 1

      Their marketing department shares a slashdot account, it's pretty standard practice, really.

      --
      moox. for a new generation.
    3. Re:Clickbait... by choprboy · · Score: 4, Informative

      I don't see anything that looks like affiliate info in that link. And you can copy/paste the URL into a new browser tab so there's no referrer info, either.

      The submitter has submitted 24 stories over the course of the last year. Every single one of the stories links to atleast one of the same two domains, on the same subject... It is pretty obviously an affiliated/sock puppet account for an employee or marketing department.

    4. Re:Clickbait... by msauve · · Score: 2

      It begins by knowing what "affiliated" means. It doesn't have to mean an Amazon-like paid referral program. It can mean that the person is an employee of, or has some other link to, the website.

      --
      "National Security is the chief cause of national insecurity." - Celine's First Law
  7. Those who forget the lessons of history.... by aaarrrgggh · · Score: 2, Interesting

    Redundancy for pumps is easy at data center scales, but the very real problem over non-trivial equipment lives is leaks.

    Way back when, we used concrete pumping hoses for chilled water to CRAC units, to facilitate relocation of the units and reduce the risk of leaks after an earthquake. Worked pretty well, they had a huge safety factor compared to concrete pressures, and they were easy to test before placing into service. Seismic performance was very predictable, and it all worked well. Then, a bad (metallurgicly) batch of fittings came in, and two or three years into operation the fittings separated from the hoses and two or three floors of data center were flooded.

    Same things have happened to copper pipes, victaulic fittings and welded connections on steel pipes, PEX pipes... everything imaginable.

    Water cooling is great, but there are a lot of weak links in the chain from a risk management perspective.

    Shame nothing ever came of HP's technology to "print" water droplets on the die and use some latent cooling in data centers.

  8. Watercooled is new? by HockeyPuck · · Score: 2

    "A lot of them grew up with PC gaming and water cooling right in their living room,"

    A lot of us grew up working on water cooled mainframes right in our own data centers.

    What's that saying about 'what's old is new again...' Now get off my lawn.

    1. Re: Watercooled is new? by merky1 · · Score: 3, Funny

      But it's a millenial telling you about it...

      --
      --WooooHoooo--
    2. Re:Watercooled is new? by Required+Snark · · Score: 1
      The Cray 1, introduced in 1976, used Freon cooling. The Freon circulated in cold bars that the individual circuit boards were mounted in. So liquid water cooling is not exactly exotic technology.

      The CRAY-1S cooling system is designed to limit the IC die temperature to a maximum of 65C. This provides a reliability margin from the 150C absolute maximum IC junction temperature. The IC package case is maintained at 54C. Heat generated in the silicon die flows through the IC package to the PC board ground plane and then to the 0.08-in thick copper cold plate. The cold plate conducts the board heat to its edges, which are held to 25C by contact with a cast aluminum cold bar. The aluminum cold bars form the twelve vertical columns in the computer mainframe into which the modules slide horizontally on 0.4-in spacings. A refrigerant, Freon 22, flows through stainless steel tubes embedded in the cold bars. The development of the composite aluminum/stainless steel cold bars represents a solution to one of the more difficult design problems of the CRAY-1. Cast aluminum is actually porous and oil mixed in with the Freon can cause reliability problems if it leaks onto the modules. A method to bond stainless steel tubing into cast aluminum had to be invented in order to make the cold bars practical.

      --
      Why is Snark Required?
  9. Don't get me started by Anonymous Coward · · Score: 0

    Today's developers, sheesh, I don't think I've ever had to work with people this dense!