Slashdot Mirror


Three-Mile-High Supercomputer Poses Unique Challenges

Nerval's Lobster writes "Building and operating a supercomputer at more than three miles above sea level poses some unique problems, the designers of the recently installed Atacama Large Millimeter/submillimeter Array (ALMA) Correlator discovered. The ALMA computer serves as the brains behind the ALMA astronomical telescope, a partnership between Europe, North American, and South American agencies. It's the largest such project in existence. Based high in the Andes mountains in northern Chile, the telescope includes an array of 66 dish-shaped antennas in two groups. The telescope correlator's 134 million processors continually combine and compare faint celestial signals received by the antennas in the ALMA array, which are separated by up to 16 kilometers, enabling the antennas to work together as a single, enormous telescope, according to Space Daily. The extreme high altitude makes it nearly impossible to maintain on-site support staff for significant lengths of time, with ALMA reporting that human intervention will be kept to an absolute minimum. Data acquired via the array is archived at a lower-altitude support site. The altitude also limited the construction crew's ability to actually build the thing, requiring 20 weeks of human effort just to unpack and install it."

45 of 80 comments (clear)

  1. Redundancy! by Joce640k · · Score: 1

    Simple answer: Have redundancy all over the place so it doesn't matter if a few modules fail. The repair crew can go in once a year and swap them.

    --
    No sig today...
    1. Re:Redundancy! by Synerg1y · · Score: 3, Insightful

      I don't think the article mentioned redundancy either day... but consider what they did: they took pre-manufactured components, hauled them up 15,000 feet and installed them... not set them up. I'm sure somewhere in this process short of hiring ALL first year grads they most likely introduced typical datacenter redundancies... load balancing, failover, arrays, etc...

      The article is about the challenges posed with operating the components at such a high altitude and for people who aren't used to high altitudes, they really can't work effectively up there limiting the pool of tech support personnel you can send up there significantly.

    2. Re:Redundancy! by sjames · · Score: 2

      Clusters don't do load balancing in the sense that a datacenter would and the nodes don't fail over (but the switches probably should in this case). If a node fails, it gets turned off and the cluster i slightly less powerful until it is replaced.

    3. Re:Redundancy! by NatasRevol · · Score: 2

      The only IT related things from the actual article are:

      Use SSDs.
      Use bigger fans.

      Seems kind of a waste to not put that in the blathering summary.

      --
      There are two types of people in the world: Those who crave closure
    4. Re:Redundancy! by NatasRevol · · Score: 3, Funny

      Great, now I need a new skill:

      IT sherpas needed for new datacenter.

      --
      There are two types of people in the world: Those who crave closure
    5. Re:Redundancy! by GameboyRMH · · Score: 2

      Or if they're having trouble finding a sysadmin, I'm available. My Spanish is decent and I have extensive experience in avoiding physical activity thus reducing the need for oxygen. Email address above!

      --
      "When information is power, privacy is freedom" - Jah-Wren Ryel
    6. Re:Redundancy! by sjames · · Score: 1

      If the hardware is in the rack, why would you idle it?

  2. Altitude Sickness... by michael_rendier · · Score: 2

    Drove three of my friends over Tioga Pass in the Sierra Nevada's in the north of Yosemite...couple of them had never been out of Louisiana...between 8000 and the summit of the pass at ~10,000 ft meant me driving while everyone else suffered from altitude sickness...the only cure is to remove to a lower elevation. Having grown up in the sierras, i was used to the elevation...but if you're not acclimated, then you're going to walk 20 feet and have to sit down to rest for 10 minutes.

    --
    There are three kinds of people in the world. Those that can count, and those that can't.
    1. Re:Altitude Sickness... by SJHillman · · Score: 1

      Mauna Loa is about 13,000ft above sea level at the peak. I was the only one that didn't have a problem breathing, but our rental SUV barely made it up... it was choking on the thin oxygen most of the last quarter of the way up. Might be one of the reasons we weren't supposed to go up the mountain according to our rental agreement.

    2. Re:Altitude Sickness... by Anonymous Coward · · Score: 3, Informative

      ...the only cure is to remove to a lower elevation.

      Top Gear suggests viagra

    3. Re:Altitude Sickness... by NoNonAlphaCharsHere · · Score: 1

      It's the fuel/air *mixture*. As you go up in altitude, you're getting a lesser volume of air, but the same amount of fuel. Modern injection systems do a better job of adjusting for air pressure, but your engine still needs to be tuned for the altitude it's going to run at. I used to live in Reno (4500 ft) and every time I drove down to San Fransisco (sea level, obviously), my car would run like crap. In town, I'd always see California cars belching smoke from all the unburned fuel in the exhaust.

    4. Re:Altitude Sickness... by Sponge+Bath · · Score: 1

      I, an out of shape old man, was able to walk around Mauna Kea (13,700 ft) without much problem as long as you move slow. 16K feet would probably put me on my ass. FYI: While visiting the area, go up Mauna Kea in the day and at night. The landscape with all the observatories is mind blowing in the sun and at night.

    5. Re:Altitude Sickness... by volxdragon · · Score: 1

      Lucky you didn't get fined by your rental car company - there are reasons those provisions are in the agreement and I believe most of the modern cars have GPS units on them. We took a tour up there about 6 years ago, going on a specially modified bus that had no problem hitting the summit (I'm also surprised they let a rental car up the final road as they limit the traffic up there to keep the dust level down which can impact the observations - I thought you had to have a sticker/permit to go up to the top)

    6. Re:Altitude Sickness... by tibit · · Score: 5, Interesting

      I think you misunderstand how it really works.

      Modern cars have air mass sensors -- they sense exactly how much air is coming into the engine, no matter what the pressure of the air is. This control is instantaneous, there's no adjustment period. The amount of fuel injected into the air is based only one the air mass and some slowly adapted tuning constants. The "lesser volume of air, same amount of fuel" assertion is completely untrue!

      So, you may ask, what if the relative partial pressure of oxygen in the air dropped with altitude -- that would be a problem, as the car only senses the air mass, not oxygen mass. It has to adapt the fuel amount relative to amount of air only based on the readings from the exaust oxygen sensor. This is not instantaneous -- the oxygen sensor readings are in effect low-pass filtered and affect the air-fuel mix very slowly, with time constants, I'd guess, on the order of an hour. Here's the good news: the relative partial pressure of oxygen stays pretty much constant at altitudes where there are roads. So it's not a problem.

      Your car has a problem of some sort, what you describe is not normal behavior.

      I was driving in a turbocharged car in the Alps and there were no performance problems related to altitude changes -- the absolute boost pressure was maintained by the ECU per throttle commands and load factor, as desired, delivering apparently same mass of air to the engine, at given load, as at sea level. This worked even on some of the highest paved roads out there. Even if there was no compressor in the intake, the engine would simply lose performance with altitude, but recover it without any undue effects when driving down.

      --
      A successful API design takes a mixture of software design and pedagogy.
    7. Re:Altitude Sickness... by tibit · · Score: 1

      Agreed. That's the key: you have to find a slow, steady pace. Going too fast makes for a very crappy day.

      --
      A successful API design takes a mixture of software design and pedagogy.
    8. Re:Altitude Sickness... by Mattcelt · · Score: 2

      This study suggests there's a real effect going on there.

      I didn't realize that Viagra is just a specific vasodilator. (It works on the smooth vessels found only in erectile tissue and in the lungs, apparently.) I wonder if there's an analogue that would work for vasoconstriction-induced migraines?

    9. Re:Altitude Sickness... by blackraven14250 · · Score: 1

      Who says he rented something even remotely new to drive up the mountain?

    10. Re:Altitude Sickness... by Anonymous Coward · · Score: 1

      Can you describe this with a car analogy?

    11. Re:Altitude Sickness... by SJHillman · · Score: 1

      This was a about 10 years ago, I don't recall needing any special sticker/permit to gain access to the road, but that could have been handled by the driver while we were at a lower altitude (we stopped at a base station of some sort). This was probably before they started adding GPS units to most rental cars. I wouldn't be surprised if that practice is less common in Hawaii anyway as there are relatively few places to take a stolen car. While our rental was sputtering along, we did encounter other smaller cars that were going up and down the road with no problem.

    12. Re:Altitude Sickness... by gkndivebum · · Score: 1

      I am going to assume that was Mauna Kea and not Mauna Loa. There is a road to the summit of the former, paved part of the way, for the people who work at the observatories up there. There is a 4WD unpaved "road" part way up the slope of Mauna Loa but no vehicular access to the summit.

      Rental car companies don't like having to come collect their vehicles from Mauna Kea after people have destroyed the brakes in them riding them all the way down the hill (or having negative interactions with invisible cows at lower elevations).

      FWIW when I make trips to the summit I bring oxygen, and spend an hour or so at the Visitor Information Station at 9300 feet before heading up to the summit.

      Working at altitude can be deceptively difficult -- an acquaintance who works at the summit describes a conversation he had which went something along the lines of "Well, I've cut it three times and it's still too short". Low PO2 isn't good for higher cognitive function.

      --
      Breathe continuously
    13. Re:Altitude Sickness... by GerryGilmore · · Score: 1

      No offense, but those friends of yours are wimps. I live in Georgia, am 59 years old, smoke cigarettes and do not exercise regularly. OTOH, I'm also not obese...However, each year for the last 3 years, my son and I have traveled to Sequoia/Kings Canyon National Park (one of the best I've ever seen, BTW) for a weeklong bout of trail hiking. Most of the trailheads are in the 7-8K ft range, with many of the trails reaching 10,000 ft and more. Certainly, we were both pausing more frequently as the elevation rose, but altitude sickness at 10,000 feet? Not a bit that either of us experienced.

    14. Re:Altitude Sickness... by Westwood0720 · · Score: 1

      Stochiometric.

      Fifteen years and I finally have a use for remembering that term. =D

    15. Re:Altitude Sickness... by michael_rendier · · Score: 1

      you didn't drive from sea level to 10,000 feet in about four hours...that's rather a bit fast to be ascending that far...it'll getcha if you do...i promise

      --
      There are three kinds of people in the world. Those that can count, and those that can't.
    16. Re:Altitude Sickness... by fotbr · · Score: 1

      Taking off in an unpressurized small plane and flying up to that altitude (granted, starting at about 400ft) in a lot less time than 4 hours (we took about 15 minutes) didn't "get" me or the other two folks I was flying with. Then again, we didn't get out and go for a walk at that altitude either.

    17. Re:Altitude Sickness... by michael_rendier · · Score: 1

      *nods*...these guys couldn't sit still...disoriented, nauseous, migraines...they had teh sickness bad...

      --
      There are three kinds of people in the world. Those that can count, and those that can't.
    18. Re:Altitude Sickness... by dadelbunts · · Score: 1

      Forced induction engines perform alot better in high altitudes due to them actively cramming air into the engine. The fact that your turbocharged vehicle did alot better in the Alps compared to his N/A SUV is no surprise.

    19. Re:Altitude Sickness... by ncc74656 · · Score: 1

      Mauna Loa is about 13,000ft above sea level at the peak. I was the only one that didn't have a problem breathing, but our rental SUV barely made it up...

      How many decades ago was that? Anything reasonably modern (and rentals are usually no more than a year or so old) would be fuel-injected and computer-controlled; altitude shouldn't affect it like that. I've driven up to Pikes Peak (about 14100', IIRC), and my car didn't have any issues. (It was a 2004 Oldsmobile Alero...nothing particularly exotic. Still drive it to work every day.)

      --
      20 January 2017: the End of an Error.
    20. Re:Altitude Sickness... by RespekMyAthorati · · Score: 1

      "Well, I've cut it three times and it's still too short"

      Ouch!

    21. Re:Altitude Sickness... by Techman83 · · Score: 1

      I know some female friends that get migraines and are prescribed viagra to control them. So it would seem this has been thought of.

      --
      # cat /dev/mem | strings | grep -i cat
      Damn, my RAM is full of cats. MEOW!!
  3. say what? by Anonymous Coward · · Score: 2, Insightful

    134 million processors, 140 kilowatts?!?

    1 miliwat per processor?

    1. Re:say what? by volxdragon · · Score: 2

      RTFA - this is not a general-purpose computer but specially built circuit boards that equate to 134M "processors" - that is why it is not eligible to be in the top supercomputer listing...

    2. Re:say what? by QuantumRiff · · Score: 2

      The processors are custom made processing boards.. they are not CPU's you would get off the shelf. They are made to do only one task.

      --

      What are we going to do tonight Brain?
  4. Re:Why process all data in place? by belthize · · Score: 2

    More than one fiber would be needed. There are 50 antennas each with multiple fibers connected to the correlator. A lot of thought went into it and despite the complications it was simpler to put the correlator there than 'down the road'.

  5. Three-mile-tall by aaron44126 · · Score: 1

    Mis-interpreted the article title and though that someone was building a supercomputer that is three miles tall. I bet that poses unique challenges too.

    1. Re:Three-mile-tall by Stele · · Score: 1

      Would be much easier to just lay it on its side.

  6. Re:Particle problems, too? by tibit · · Score: 1

    The hard drives don't belong at that altitude, then. The correlators can well be completely diskless machines, even without a solid state drive. They can boot over the network from the lower-altitude server station.

    --
    A successful API design takes a mixture of software design and pedagogy.
  7. Re:3 mile high supercomputer! by Andrio · · Score: 1

    That's exactly what I thought.

    "A three mile tall supercomputer? You're damned right that poses some unique challenges!"

    --
    The Internet King? I wonder if he could provide faster nudity.
  8. Re:Particle problems, too? by ls671 · · Score: 1

    This is ridiculous. Just pressurize the server room or whole building and be done with it. That layer of air would automagically reappear for the heads to glide over the platters.

    As for particles, I know nothing about the subject but I guess that mountain isn't much closer than we are from the particle sources. I do not think the additional layer of atmosphere said particles have to go through to get to us makes a difference.

    My understanding is that particles are deviated at higher altitude than mountains by the Earth magnetic field.

    --
    Everything I write is lies, read between the lines.
  9. A/C Strikes Again by Anonymous Coward · · Score: 1

    Any discussion in TFA about why they aren't just sending that data over optical cables to wherever would make sense to house a data-center? e.g. ground-level?

    Why does it need to be so near the array?

    Speed of signal over fibre can't be the difference (i.e. what difference does 1 km or 40km make? not much).

    1. Re:A/C Strikes Again by snspdaarf · · Score: 1

      e.g. ground-level?

      Whoa! A levitating three mile high supercomputer!

      --
      Why, without your clothes, you're naked, Miss Dudley!
  10. I Couldn't Help But Think of a Better Location by LifesABeach · · Score: 1

    The far side of the Moon. No clouds, no rain, and only a little bit of dust every so often.

  11. Re:Particle problems, too? by ls671 · · Score: 1

    At that altitude, the pressure is about half that of sea level. So to pressurize the building, you would have to do so by several psi to bring it down to something a little lower.

    well, 1/2 atmosphere = 7 psi.

    5 psi should be enough,

    8 psi causes stress on en airplane fuselage but planes have to be light. Submarines can stand 1,500 psi and above...

    My point is that it just might end up being cheaper than fixing/adapting every piece of equipment individually, in a never ending quest. Who know?

    --
    Everything I write is lies, read between the lines.
  12. Put the computer at a lower altitude. by jklovanc · · Score: 1

    They are combining signals from dishes separated by up to 16km so it is not a necessity that the supercomputer be right next to one of the dishes. Why not build the supercomputer at the base of the mountain instead of the top. They are already beaming raw data around so it will not make a difference.

  13. Virtualization! by Sigg3.net · · Score: 1

    Just virtualize the supercomputer in the clouds and put the virtual machine on the mountain!

    See?

  14. Re:Particle problems, too? by Jonathan_S · · Score: 1

    This is ridiculous. Just pressurize the server room or whole building and be done with it. That layer of air would automagically reappear for the heads to glide over the platters.

    Of course that means that the room pressurization would be a single point of failure for every hard drive you had. Lose pressure and every drive suffers a head crash simultaneously... Oops.

    If I didn't need the storage volume I'd certainly prefer drives like SSDs that didn't require pressurization to work at that altitude. One less thing to go wrong. (Although the ability to pressurize the building when necessary to make maintenance / upgrades easier on the IT guys would be cool and useful)