Slashdot Mirror

← Back to Stories (view on slashdot.org)

Managing the Most Remote Data Center In the World

Posted by kdawson on from the talk-about-latency dept.

blackbearnh writes "Imagine that your data center was in the most geographically remote location in the world. Now imagine that you can only get to it 4 months of the year. Just for fun, add in some of the most extreme weather conditions in the world. That's the challenge that faces John Jacobsen, one of the people responsible for making sure that the data from the IceCube Neutrino Observatory makes it all the way from the South Pole to researchers across the world. In an interview recorded at OSCON, Jacobsen talks about the problems that he has to face (video), which includes (surprisingly) keeping the data center cool. If you're ever griped because you had to haul yourself across town in the middle of the night to fix a server crash, this interview should put things in perspective."

14 of 98 comments (clear)

  1. So it's possible after all... by Drakkenmensch · · Score: 4, Funny

    ... to sell air conditionning in the south pole?

    1. Re:So it's possible after all... by Drakkenmensch · · Score: 5, Funny

      But then a bunch of penguins could sneak in and replace all their copies of Windows 2008 server with Linux. Wait, this plan actually sounds better and better every second...

  2. Unfortunately... by fuzzyfuzzyfungus · · Score: 4, Funny

    The video makes it impossible to tell if this guy is the real thing, or if The Thing has had a change to catch him in the cold isle and duplicate him. I fear to imagine what it would be capable of once it uses the base's internet connection to discover tentacle-rape hentai...

  3. Re:Kdawson . . . by rossdee · · Score: 5, Funny

    I think you're almost certain to have to fix problems in the middle of the night down there, after all they are about 6 months long...

  4. Remote, But Not Remotest by Iskender · · Score: 4, Interesting

    The South Pole is the remotest by many standards in kilometres. However, I recall some recent research which came to the conclusion that parts of the Himalayas are the remotest on Earth. At least some parts of the year you can basically just fly to the South Pole. Not so much with the worst parts of the Himalayas - I seem to recall a minimum travel time of one or two weeks.

    There was an article on the research on the BBC site about this, but it's fiendishly hard to find. Plus points to anyone who can dig it up.

    Oh and I should avoid sounding cynical and say that the stuff in the article is certainly a cool challenge. It's still a tricky location compared to 95% of all other land, and I'd love to work on problem-solving like that myself.

    1. Re:Remote, But Not Remotest by mc1138 · · Score: 4, Insightful

      Yes but last I checked there was no data center there... The article says the most remote data center on Earth, not the most remote spot. Plus I might put the bottom of certain parts of the ocean at even more remote than the Himalayas, as there are spots down there no one's ever reached.

      --
      The musings of just another geek and his junk.
    2. Re:Remote, But Not Remotest by Anonymous Coward · · Score: 3, Interesting

      Agreed, this is not remote because you can if you need to take a vehicle all the way there. Plane/Boat/snow machine.. Top of Everest is remote as it takes a week of hiking to get there.

      Me I have a remote "datacenter" to manage.. It's atop a 200 foot tower in northern Michigan... it's at least a 4 hour drive + 1/2 hour hike and then you have to climb 200 feet vertical with all your gear.

      Luckily it has not needed to be touched for 4 years. my remote administration over ham radio frequencies has worked very well. This guy has it easy with speeds over 1200bps and constant communication. try lag times measuring in seconds and 1200bps to manage a linux box from 250 miles away across 5 digipeaters. It's a weather station and Digipeater with a mailbox. Runs a pre 2.0 kernel and is 100% unhackable unless you try to brute force the password (no you cant simply use the password I sent in the clear... It uses single use passwords and I'm only 1/2 way through the list.

      I'll give this guy the Uber geek crown if he adds a 200 foot vertical climb to his trip to the "datacenter". Because kids, that alone is a major PITA... nothing like screwing around with a laptop 200 feet up, in wind and having the damn thing sway about 2 feet back and forth while you work. Oh... dont drop anything... during setup when the RF guys screwed up and cut my comms wire to the base of the tower I dropped a laptop from 200 feet... Even Panasonic toughbooks dont survive that fall.

    3. Re:Remote, But Not Remotest by dargaud · · Score: 3, Interesting

      I'll give this guy the Uber geek crown if he adds a 200 foot vertical climb to his trip to the "datacenter"

      Well, let's see if I can take that crown: I managed a bunch of experiments (and their associated computers and comm systems) at Dome C in 2005. It's higher than South Pole. And colder to boots (we had -78C). And some instruments were atop a 100ft tower (now raised to 200ft) were it was windy as hell in addition to being as cold as stated. In winter it was physically next to impossible to climb: hard-packed snow covered the scale (you had to clean every step), your breath would freeze your clothing solid around your head, your glasses would fog in a few seconds turning you blind, and if you exposed a blip of skin it would feel like a knife went through it immediately. Gosh, I miss that place: the view was fantastic.

      --
      Non-Linux Penguins ?
  5. Space probes by Honken · · Score: 5, Insightful

    What I find even more impressive is how NASA, ESA and others manages space probes I think, that's really extreme conditions in every way. Often huge communication delays and poor bandwidth, absolutely no chance of at least eventually fixing a problem on-site, hardware constantly being subjected to intense radiation and extreme temperature differences. Imagine that rather unpleasant feeling you get when you reboot a remote server and you know you won't be able to go on-site any time soon to fix it if you did something wrong, then take that feeling and add the fact that you can _never_ fix it, that it costs millions or even billions to send it there, that lots of valuable science might be lost or never take place, and that you'll be guaranteed to read about your mistake in the news the following day. I guess it calls for rather extreme levels of testing before doing any changes at all.

    1. Re:Space probes by Cornwallis · · Score: 4, Interesting

      I remember when the LEMs took off from the moon on the later missions and some comm guy at NASA was able to track the liftoff with the video camera left on the moon. The idea that he was able to *anticipate* the liftoff and ascent and remotely track still stands out as one of the all time cool things to watch.

    2. Re:Space probes by dargaud · · Score: 4, Informative

      What I find even more impressive is how NASA, ESA and others manages space probes I think, that's really extreme conditions in every way.

      Antarctica can be meaner in several ways: - you don't have a direct line of communication with the rest of the world (space probes do). Hell, you can't even have a direct comm with geosync satellites. - water ! Take thin crystals of ice, add lots of wind and you end up with water deep inside even sealed boxes; hence shorts and very quick rusting of components. - temperature changes. In space you surround your satellite with some heat conductive sheets and the temp basically never changes (unless you go into the earth shadow). In antarctica you can have -80C in winter, -10 in summer. To say nothing that the cold has unforeseen effects on materials (dielectric changes, materials becoming brittle...) - unstable power: the power comes from big diesel generators and is shared between experiments, people, etc... It goes out, the temp of the room where your computer is falls to -60 in 15 minutes. Power comes back, computer tries to boot. Bye, bye hard drive. - budget: experiments for Antarctica have much less than 1/10 the budget of equivalent space experiments. And most of it is eaten by logistics. So you end up with standard computers and a few hack and a guy standing nearby (me) to kick it if necessary.

      --
      Non-Linux Penguins ?
  6. You don't need to go as far by mseeger · · Score: 3, Informative

    I don't envy someone with a job like that. It is already very difficult to serve locations less remote. E.g. to offer a SLA for a network which spans over 30 locations in the world, we have to make sure that spare parts arrive around the world within a well defined time frame. We don't want anything fancy: a week would be finde by us. But i haven't found any transportation company that guarantees delivery on site in 3rd world countries (big cities) within that time frame. All make exceptions like "customs", which doesn't help me.

    CU, Martin

    1. Re:You don't need to go as far by mseeger · · Score: 3, Informative

      Or just blow the budget and have a spare set of equipment at each location. When something dies, take out the replacement that is already there, then worry about shipping a new replacement.

      According to the customer, this has been tried and failed... Unused spare equipment has a tendency to vanish.

      CU, Martin

    2. Re:You don't need to go as far by mseeger · · Score: 3, Interesting

      Yes, it is expensive, and yes, it's a pain to have to store and keep track of the spares inventory (which always seems to 'go walkies'), but if you have a demanding SLA, it's the only way to go.

      Expensive is not the problem, getting the walkies is... According to customer, they experience a strange "honor code" in third world countries... Operative systems seem to be quite safe, but any spare equipment is fair game (Africa is the biggest problem in this regard). Trying to fool the people by faking spare systems to be operative was also not successful.

      CU, Martin