Managing the Most Remote Data Center In the World

← Back to Stories (view on slashdot.org)

Managing the Most Remote Data Center In the World

Posted by kdawson on Friday July 23, 2010 @12:49AM from the talk-about-latency dept.

blackbearnh writes "Imagine that your data center was in the most geographically remote location in the world. Now imagine that you can only get to it 4 months of the year. Just for fun, add in some of the most extreme weather conditions in the world. That's the challenge that faces John Jacobsen, one of the people responsible for making sure that the data from the IceCube Neutrino Observatory makes it all the way from the South Pole to researchers across the world. In an interview recorded at OSCON, Jacobsen talks about the problems that he has to face (video), which includes (surprisingly) keeping the data center cool. If you're ever griped because you had to haul yourself across town in the middle of the night to fix a server crash, this interview should put things in perspective."

98 comments

Min score:

Reason:

Sort:

So it's possible after all... by Drakkenmensch · 2010-07-23 00:50 · Score: 4, Funny

... to sell air conditionning in the south pole?
1. Re:So it's possible after all... by elrous0 · 2010-07-23 01:46 · Score: 1
  
  Wouldn't it be cheaper just to put in a window?
  
  --
  SJW: Someone who has run out of real oppression, and has to fake it.
2. Re:So it's possible after all... by somersault · 2010-07-23 02:22 · Score: 1
  
  Probably doesn't help having your data center full of snow..
  
  --
  which is totally what she said
3. Re:So it's possible after all... by Anonymous Coward · 2010-07-23 02:36 · Score: 0
  
  I worked at Admundson-Scott South Pole Station... in the old dome in 2000/2001. I worked in the network admin area. We opened the window to ambient when it got too hot, ambient within the dome being -60.
  I would buy my beer warm and put it on the floor, in the corner by my desk. There was a 3 inch layer of ice than ran all along the room at the meeting of the wall and the floor. By my 12 hour shift in, the DB Beer from NZ was perfect.
  Great memories...
4. Re:So it's possible after all... by Anonymous Coward · 2010-07-23 03:04 · Score: 0
  
  Surely there is some way to conduct cold in (say, oh, a refrigerant liquid that won't freeze solid at their extremes), or heck, given how cold it is out there, just maybe some metal to conduct the cold in, and wick the heat out.
5. Re:So it's possible after all... by somersault · 2010-07-23 03:18 · Score: 1
  
  Technically it's the heat conducting and not the cold, but I do like your idea better!
  
  --
  which is totally what she said
6. Re:So it's possible after all... by Drakkenmensch · 2010-07-23 03:39 · Score: 5, Funny
  
  But then a bunch of penguins could sneak in and replace all their copies of Windows 2008 server with Linux. Wait, this plan actually sounds better and better every second...
7. Re:So it's possible after all... by kiwimn · 2010-07-23 03:53 · Score: 1
  
  As a kiwi living in the US I would happily work 12 hour shifts in the middle of Antarctica if it meant getting a few DBs once in a while!
8. Re:So it's possible after all... by Anonymous Coward · 2010-07-23 04:18 · Score: 0
  
  There's an old joke about that - punchline is you can have sex more easily at -20 than at -60. Oh wait, those were freezers...
9. Re:So it's possible after all... by Deep+Penguin · 2010-07-23 05:16 · Score: 1
  
  I worked in the same building in 03/04 and 05/06, though the last few months it was just to clear stuff before they demolished it. The commute from my room upstairs was nice and short - a bonus.
  As cramped as it was, it was still great to work there.
10. Re:So it's possible after all... by kievit · 2010-07-24 11:50 · Score: 1
  
  Actually practically all IceCube servers at the Pole, running the data acquisistion, processing & filtering are running linux. So those penguins would not have much to do, except join the party. :-)
  (And a nitpick: IceCube is actually at the geographic South Pole, too far away from the Antarctic coast for any penguin to reach it.)
Probably doesn't seem that cold ... by Anonymous Coward · 2010-07-23 00:54 · Score: 1, Funny

... to the Wookie who was interviewing him.
Re:Kdawson . . . by Allicorn · 2010-07-23 00:59 · Score: 1

Indeed. "Gripped"? WTH?

--
OMG!!! Ponies!!!
Unfortunately... by fuzzyfuzzyfungus · 2010-07-23 01:01 · Score: 4, Funny

The video makes it impossible to tell if this guy is the real thing, or if The Thing has had a change to catch him in the cold isle and duplicate him. I fear to imagine what it would be capable of once it uses the base's internet connection to discover tentacle-rape hentai...
1. Re:Unfortunately... by ctchristmas · 2010-07-23 02:01 · Score: 1
  
  Luckily, it would probably be destroyed once it stumbled across goatse.
2. Re:Unfortunately... by laejoh · 2010-07-23 07:17 · Score: 1
  
  Why tentacle-rape hentai when you have elder things? Tekeli li, tekeli li!
Re:Kdawson . . . by rossdee · 2010-07-23 01:02 · Score: 5, Funny

I think you're almost certain to have to fix problems in the middle of the night down there, after all they are about 6 months long...
Re:Kdawson . . . by Anonymous Coward · 2010-07-23 01:05 · Score: 0

According to kdawson, you do the gripping, you don't get gripped. As to who or what you grip, I'm not sure.
is this real? by Anonymous Coward · 2010-07-23 01:05 · Score: 0

lol... the interviewer is like humpty dumpty
Notice how Jacobsen keeps glancing over at him? by Anonymous Coward · 2010-07-23 01:07 · Score: 0

I think he's afraid he's going to be eaten.
Remote, But Not Remotest by Iskender · 2010-07-23 01:08 · Score: 4, Interesting

The South Pole is the remotest by many standards in kilometres. However, I recall some recent research which came to the conclusion that parts of the Himalayas are the remotest on Earth. At least some parts of the year you can basically just fly to the South Pole. Not so much with the worst parts of the Himalayas - I seem to recall a minimum travel time of one or two weeks.
There was an article on the research on the BBC site about this, but it's fiendishly hard to find. Plus points to anyone who can dig it up.
Oh and I should avoid sounding cynical and say that the stuff in the article is certainly a cool challenge. It's still a tricky location compared to 95% of all other land, and I'd love to work on problem-solving like that myself.
1. Re:Remote, But Not Remotest by mc1138 · 2010-07-23 01:11 · Score: 4, Insightful
  
  Yes but last I checked there was no data center there... The article says the most remote data center on Earth, not the most remote spot. Plus I might put the bottom of certain parts of the ocean at even more remote than the Himalayas, as there are spots down there no one's ever reached.
  
  --
  The musings of just another geek and his junk.
2. Re:Remote, But Not Remotest by dimethylxanthine · 2010-07-23 01:21 · Score: 0
  
  "Remotest" is only a matter of perspective.
  
  -----
  Zen.
3. Re:Remote, But Not Remotest by Iskender · 2010-07-23 01:56 · Score: 1
  
  You're basically right, but the summary still starts by talking about the most remote place, and that's what most will read anyway.
4. Re:Remote, But Not Remotest by Anonymous Coward · 2010-07-23 02:07 · Score: 3, Interesting
  
  Agreed, this is not remote because you can if you need to take a vehicle all the way there. Plane/Boat/snow machine.. Top of Everest is remote as it takes a week of hiking to get there.
  Me I have a remote "datacenter" to manage.. It's atop a 200 foot tower in northern Michigan... it's at least a 4 hour drive + 1/2 hour hike and then you have to climb 200 feet vertical with all your gear.
  Luckily it has not needed to be touched for 4 years. my remote administration over ham radio frequencies has worked very well. This guy has it easy with speeds over 1200bps and constant communication. try lag times measuring in seconds and 1200bps to manage a linux box from 250 miles away across 5 digipeaters. It's a weather station and Digipeater with a mailbox. Runs a pre 2.0 kernel and is 100% unhackable unless you try to brute force the password (no you cant simply use the password I sent in the clear... It uses single use passwords and I'm only 1/2 way through the list.
  I'll give this guy the Uber geek crown if he adds a 200 foot vertical climb to his trip to the "datacenter". Because kids, that alone is a major PITA... nothing like screwing around with a laptop 200 feet up, in wind and having the damn thing sway about 2 feet back and forth while you work. Oh... dont drop anything... during setup when the RF guys screwed up and cut my comms wire to the base of the tower I dropped a laptop from 200 feet... Even Panasonic toughbooks dont survive that fall.
5. Re:Remote, But Not Remotest by RKThoadan · 2010-07-23 02:12 · Score: 1
  
  We actually have sent manned bathyspheres and ROVs to the bottom of the Challenger deep, so we can get there. Still, while the remoteness might be debatable, it's easily the most dangerous environment human's have ever been to. As for mountain peaks, we've landed helicopters on Everest's peak, so I'd guess we could land one on any other peak if we really need to.
  Challenger Deep: http://en.wikipedia.org/wiki/Challenger_deep#Descents
  Everest: http://www.nationalgeographic.com/adventure/0509/whats_new/helicopter_everest.html
6. Re:Remote, But Not Remotest by Anonymous Coward · 2010-07-23 02:19 · Score: 0
  
  The headline says "remotest data center" I was going to go for the obligitory didn't RTFS joke, but seriously; "remotest"? I never heard that word before I came to Slashdot.
7. Re:Remote, But Not Remotest by drsmithy · 2010-07-23 02:31 · Score: 1
  
  I'll give this guy the Uber geek crown if he adds a 200 foot vertical climb to his trip to the "datacenter". Because kids, that alone is a major PITA... nothing like screwing around with a laptop 200 feet up, in wind and having the damn thing sway about 2 feet back and forth while you work. Oh... dont drop anything... during setup when the RF guys screwed up and cut my comms wire to the base of the tower I dropped a laptop from 200 feet... Even Panasonic toughbooks dont survive that fall.
  Why isn't the system at the bottom of the tower with 200 feet of cable above it ?
8. Re:Remote, But Not Remotest by Deep+Penguin · 2010-07-23 02:36 · Score: 1
  
  Having done so several times, yes, between about late-October/early-November and mid-February, you can "just fly" to the South Pole. The trip can take as little as 5 days, but 8-10 days is more ordinary. I've been around for periods in the summer where nothing came and went from McMurdo (the logistics hub at the coast, and one node on the trip) for 10 days in a row (and that's not the record).
  So for 1/3 of the year, you can get on a succession of airplanes and, weather permitting, get to the Pole in 1-2 weeks. For 2/3 of the year, it's a major, major undertaking to get there (try landing a plane at -85F in the dark... they did that in April, 2001; the other "mid-winter rescues" were after the sun was up but before the regular summer season started).
  Even once you are there, it's not exactly a stroll in the park - the data center is a hike out from the main station (15 minutes when it's light, a lot longer in the dark). Yes, we walk. In the winter it gets too cold for weeks at a time to safely operate machinery (below -60F it had better be important, below -85F it had better be an emergency).
9. Re:Remote, But Not Remotest by Anonymous Coward · 2010-07-23 02:42 · Score: 0
  
  because that wouldn't make for nearly as interesting a story.
  not _all_ apparently ass-backward solutions are the result of poor planning.
10. Re:Remote, But Not Remotest by dargaud · 2010-07-23 02:45 · Score: 3, Interesting
  
  I'll give this guy the Uber geek crown if he adds a 200 foot vertical climb to his trip to the "datacenter"
  Well, let's see if I can take that crown: I managed a bunch of experiments (and their associated computers and comm systems) at Dome C in 2005. It's higher than South Pole. And colder to boots (we had -78C). And some instruments were atop a 100ft tower (now raised to 200ft) were it was windy as hell in addition to being as cold as stated. In winter it was physically next to impossible to climb: hard-packed snow covered the scale (you had to clean every step), your breath would freeze your clothing solid around your head, your glasses would fog in a few seconds turning you blind, and if you exposed a blip of skin it would feel like a knife went through it immediately. Gosh, I miss that place: the view was fantastic.
  
  --
  Non-Linux Penguins ?
11. Re:Remote, But Not Remotest by Anonymous Coward · 2010-07-23 02:47 · Score: 0
  
  I think you're talking about this (pics).
  I'm a bit confused as to what it's what you meant, because while it exactly answers the description of "research based on 'remotest' as defined by time/difficulty of reaching it, rather than plain kilometres, and concludes that the Himalayas wins", it doesn't match "fiendishly hard to find", being the second google result for "remotest place on earth".
  Anyway, someone sling a +1 interesting to this poor AC, who's just lost it thanks to me posting this reply.
12. Re:Remote, But Not Remotest by Muad'Dave · 2010-07-23 03:16 · Score: 2, Informative
  
  Attenuation, plain and simple. At 145MHz, the attenuation of 200' of ok coax (LMR-400 @ $0.86/ft) is almost exactly 3dB - that's 50% of your transmitted power wasted heating the coax. Additionally that makes your receiver 3dB more deaf to weak signals.
  At 450MHz, it's even worse. Attenuation is 5.4dB, which steals about 71.3% of your signal.
  Stepping up to 1/2" Andrews DF4-50A heliax, you find the price jumps to $1.69/ft but the attenuation at 450MHz drops to only 2.9dB for 200'. We're still losing 50% of our signal.
  Stepping up again to 7/8" hardline (3.99/ft), attenuation is 1.44dB @ 450 MHz - you _still_ lose 30% of the signal.
  By putting the transmitter, computer, etc at the top of the tower, you get the best of all worlds - very low attenuation in the RF path, no long computer cables to pick up stray RF or lightning-induced surges, and fewer vandals. The downside is that you have to run power up the tower (usually 120VAC, since running 12VDC hits the same sorts of snags as coax - lots and lots of loss. Imagine 200' of welding cable - $$$ and heavy!). As long as you have a decent AC filter up the tower, you're safer than separating the components. The downside, however, is a beastly climb if anything fails catastrophically.
  
  --
  Tiller's Rule: Never use a word in written form that you've only heard and never read. You will end up looking foolish.
13. Re:Remote, But Not Remotest by Muad'Dave · 2010-07-23 03:20 · Score: 1
  
  can you say "Winch"? I'd submit to being hauled up on a cable to the top of the tower before going through all that! I've done plenty of tower work, but always in safe, warm (sometimes hot), low-wind conditions. Rohn 25 towers are scary to climb - you look down and you can't see the tower because your feet block the view. Towers look solid from the ground, but boy do they sway when you're 125' in the air!
  
  --
  Tiller's Rule: Never use a word in written form that you've only heard and never read. You will end up looking foolish.
14. Re:Remote, But Not Remotest by tehcyder · 2010-07-23 03:22 · Score: 1
  
  Top of Everest is remote as it takes a week of hiking to get there
  
  Couldn't you just fly up in a helicopter and jump out somewhere near the top?
  
  No, I have no idea what I'm talking about.
  
  --
  To have a right to do a thing is not at all the same as to be right in doing it
15. Re:Remote, But Not Remotest by Monty_Lovering · 2010-07-23 03:57 · Score: 1
  
  One could argue that the bottom of the ocean is not _on_ earth
16. Re:Remote, But Not Remotest by mc1138 · 2010-07-23 04:46 · Score: 1
  
  Actually no, I don't see how that's possible.
  
  --
  The musings of just another geek and his junk.
17. Re:Remote, But Not Remotest by Haffner · 2010-07-23 04:51 · Score: 1
  
  IIRC there was a slashdot article recently about the deepest caves in the world... and many are still unexplored. I would throw these into the running for "most remote location on Earth" as well.
  
  --
  "Going to war without the French is like going deer hunting without your accordion." ~General Norman Schwarzkopf
18. Re:Remote, But Not Remotest by Anonymous Coward · 2010-07-23 04:59 · Score: 0
  
  Two landings (by the same person) have been done at the Everest peak.
19. Re:Remote, But Not Remotest by Monty_Lovering · 2010-07-23 05:24 · Score: 2, Funny
  
  Silly jokes are not worth explaining. Next time I will employ sub(marine)titles for the hard of understanding.
20. Re:Remote, But Not Remotest by Anonymous Coward · 2010-07-23 05:52 · Score: 0
  
  I don't know if I'd feel all that great about trusting a winch + cable that's been sitting in the Antarctic for more than about 30 minutes...
21. Re:Remote, But Not Remotest by evilviper · 2010-07-23 06:11 · Score: 1
  
  I recall some recent research which came to the conclusion that parts of the Himalayas are the remotest on Earth. At least some parts of the year you can basically just fly to the South Pole.
  Parts of the Himalayas may be difficult to physically reach, but they're not as bad off in many ways. Numerous orbiting communications satellites, for one thing, work everywhere on the planet EXCEPT for the poles.
  
  --
  Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
22. Re:Remote, But Not Remotest by Golddess · 2010-07-23 06:45 · Score: 1
  
  I can't imagine why. It's a perfectly cromulent word that embiggens us all.
  
  --
  "I'm not sure I like the fugnutish tone you used in your post!" -RogL (608926)-
23. Re:Remote, But Not Remotest by X0563511 · 2010-07-23 07:08 · Score: 1
  
  So... what's wrong with fiber?
  
  --
  For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
24. Re:Remote, But Not Remotest by X0563511 · 2010-07-23 07:09 · Score: 1
  
  [quote]Ahem, did you get lost ?
  404 Error — Not found[/quote]
  
  --
  For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
25. Re:Remote, But Not Remotest by Golddess · 2010-07-23 07:28 · Score: 1
  
  Maybe GP means "It's not on Earth, man, it's like, in it!"?
  
  --
  "I'm not sure I like the fugnutish tone you used in your post!" -RogL (608926)-
26. Re:Remote, But Not Remotest by StikyPad · 2010-07-23 07:46 · Score: 1
  
  Fixed that for you.
  
  --
  https://www.eff.org/https-everywhere
27. Re:Remote, But Not Remotest by Muad'Dave · 2010-07-23 07:49 · Score: 1
  
  It's very hard to transmit Radio Frequency energy over fiber. :-)
  You could use fiber for the control and data signals, but then you'd still have to run power up for the transmitter, and your baseband signal would also have to be sent up that way. Audio could be modulated directly onto the fiber or digitized, but you'd still need an awful lot of (custom?) hardware up the pole to do the light->electronic->light translations. With all that up the pole, why not stick it all up there? That way you don't need a secure 'shack' at the bottom of the pole.
  It sounds like the original poster is doing AX.25, which is 1200 baud FSK over a voice width channel. You can either do AFSK (using audio tones to modulate an FM or PM transmitter), or 'real' FSK, where the actual carrier is modulated. Most TNCs (terminal node controllers) do the audio deal - they take RS-232 in and emit the right tones for over the air and vice versa. An RS-232->fiber->RS-232 converter setup would work, but now you've got the TNC and transmitter up the pole, and you'll need a safe ground-level place for the PC.
  It always comes down to tradeoffs, doesn't it?
  
  --
  Tiller's Rule: Never use a word in written form that you've only heard and never read. You will end up looking foolish.
28. Re:Remote, But Not Remotest by Iskender · 2010-07-23 08:12 · Score: 1
  
  it doesn't match "fiendishly hard to find", being the second google result for "remotest place on earth".
  Since we aren't robots, there are always lots of options. I tried many, but always used "location" which to my brain sounds like a more appropriate formal word here.
  Thanks for the link.
29. Re:Remote, But Not Remotest by Carnildo · 2010-07-23 08:15 · Score: 1
  
  -78 C? I don't think I'd trust anything with moving parts in those conditions. Materials don't behave the way you expect in extreme cold: steel cables become brittle, lubricants freeze and shatter, thermal contraction messes with tolerances, and everything gets coated with ice.
  
  --
  "They redundantly repeated themselves over and over again incessantly without end ad infinitum" -- ibid.
30. Re:Remote, But Not Remotest by sznupi · 2010-07-23 08:59 · Score: 1
  
  Since a helicopter landed on the Everest few years back, I imagine you can also basically just fly to the worst parts of the Himalayas, if you really want to.
  
  --
  One that hath name thou can not otter
31. Re:Remote, But Not Remotest by Jeff+DeMaagd · 2010-07-23 10:10 · Score: 1
  
  Did you miss the headline? Is there a data center in the Himalayas? If not, then it's not in the running for "most remote data center".
  Besides, a location that you can't even get near for half the year seems to win.
32. Re:Remote, But Not Remotest by Macrat · 2010-07-23 10:18 · Score: 1
  
  So... what's wrong with fiber?
  Most people just don't get enough fiber.
33. Re:Remote, But Not Remotest by clemenstimpler · 2010-07-23 10:25 · Score: 1
  
  http://www.visualcomplexity.com/vc/project_details.cfm?id=672&index=672&domain
  Search terms: remotest travel time week bbc - you find it on the second page. How do I receive my Google-Fu-points? ;)
34. Re:Remote, But Not Remotest by dargaud · 2010-07-24 20:18 · Score: 1
  
  -78 C? I don't think I'd trust anything with moving parts in those conditions. Materials don't behave the way you expect in extreme cold: steel cables become brittle, lubricants freeze and shatter, thermal contraction messes with tolerances, and everything gets coated with ice.
  Correct. We stopped using snowmachines below -55C. And the only mechanical thing we kept using outdoors was the (Caterpilar) loader for the snow melter (to produce water). Even the soles of our special shoes would turn hard as rock below that. Sorry about the bad link in the previous post.
  
  --
  Non-Linux Penguins ?
35. Re:Remote, But Not Remotest by tbird81 · 2010-07-25 07:28 · Score: 1
  
  I can't imagine why. It's a perfectly cromulent word that embiggens us all.
  I really have to refudiate that statement.
grammar by Anonymous Coward · 2010-07-23 01:10 · Score: 0

Remotest? In the headline? Really?
1. Re:grammar by Anonymous Coward · 2010-07-23 01:52 · Score: 2, Informative
  
  Grammar nazi fail:
  remote
  - 5 dictionary results
  remote /rmot/ Show Spelled [ri-moht] Show IPA adjective,-moter, -motest , noun
  –adjective
  1.
  far apart; far distant in space; situated at some distance away: the remote jungles of Brazil.
  2.
  out-of-the-way; secluded: a remote village; a remote mountaintop.
  3.
  distant in time: remote antiquity.
  4.
  distant in relationship or connection: a remote ancestor.
  5.
  operating or controlled from a distance, as by remote control: a remote telephone answering machine.
  6.
  far off; abstracted; removed: principles remote from actions.
  7.
  not direct, primary, or proximate; not directly involved or influential: the remote causes of the war.
  8.
  slight or faint; unlikely: not the remotest idea; a remote chance.
  9.
  reserved and distant in manner; aloof; not warmly cordial.
2. Re:grammar by blackbearnh · 2010-07-23 02:28 · Score: 1
  
  I'll take credit for the gripped vs griped issue (I'll blame it on the fact I posted it past midnight, 3 timezones west of my normal). But if you check Firehose, the story started out with the headline of "Most Remote". And for the record, I'm neither a Wookie, nor related to Humpty Dumpty. I'm a Ninja Coder of the Kung Fu Panda school.
Space probes by Honken · 2010-07-23 01:11 · Score: 5, Insightful

What I find even more impressive is how NASA, ESA and others manages space probes I think, that's really extreme conditions in every way. Often huge communication delays and poor bandwidth, absolutely no chance of at least eventually fixing a problem on-site, hardware constantly being subjected to intense radiation and extreme temperature differences. Imagine that rather unpleasant feeling you get when you reboot a remote server and you know you won't be able to go on-site any time soon to fix it if you did something wrong, then take that feeling and add the fact that you can _never_ fix it, that it costs millions or even billions to send it there, that lots of valuable science might be lost or never take place, and that you'll be guaranteed to read about your mistake in the news the following day. I guess it calls for rather extreme levels of testing before doing any changes at all.
1. Re:Space probes by fuzzyfuzzyfungus · 2010-07-23 01:16 · Score: 1
  
  Plus, having all the usual cooling methods not work at all would be a bit of a downer(ok, yeah, normal servers do lose some tiny amount of energy by radiation, so I guess that counts).
  
  No conductive cooling, you are floating in the depths of space, surrounded by nothing.
  
  No convection. There isn't any atmosphere, nor any gravity(of use, obviously gravitation is universal)
  
  Even in sealed capsules with humans, forced air just moves the problem around, there isn't nearly enough air to treat it as an arbitrarily deep heatsink.
  
  The ability to make up for several hundreds or thousands of watts of heat dissipation just by pointing a few cheap fans at something is really a huge luxury...
2. Re:Space probes by Cornwallis · 2010-07-23 01:28 · Score: 4, Interesting
  
  I remember when the LEMs took off from the moon on the later missions and some comm guy at NASA was able to track the liftoff with the video camera left on the moon. The idea that he was able to *anticipate* the liftoff and ascent and remotely track still stands out as one of the all time cool things to watch.
3. Re:Space probes by dargaud · 2010-07-23 02:53 · Score: 4, Informative
  
  What I find even more impressive is how NASA, ESA and others manages space probes I think, that's really extreme conditions in every way.
  Antarctica can be meaner in several ways: - you don't have a direct line of communication with the rest of the world (space probes do). Hell, you can't even have a direct comm with geosync satellites. - water ! Take thin crystals of ice, add lots of wind and you end up with water deep inside even sealed boxes; hence shorts and very quick rusting of components. - temperature changes. In space you surround your satellite with some heat conductive sheets and the temp basically never changes (unless you go into the earth shadow). In antarctica you can have -80C in winter, -10 in summer. To say nothing that the cold has unforeseen effects on materials (dielectric changes, materials becoming brittle...) - unstable power: the power comes from big diesel generators and is shared between experiments, people, etc... It goes out, the temp of the room where your computer is falls to -60 in 15 minutes. Power comes back, computer tries to boot. Bye, bye hard drive. - budget: experiments for Antarctica have much less than 1/10 the budget of equivalent space experiments. And most of it is eaten by logistics. So you end up with standard computers and a few hack and a guy standing nearby (me) to kick it if necessary.
  
  --
  Non-Linux Penguins ?
4. Re:Space probes by StuartHankins · 2010-07-23 03:27 · Score: 1
  
  Mod +1 Informative please; the link is informative as well.
5. Re:Space probes by Anonymous Coward · 2010-07-23 06:06 · Score: 0
  
  Then imagine a ground station in the Antarctica being used to trouble shoot a satellite.
  I haven't seen any official press about it so I cannot divulge too much information, but this is the current situation for one of the ESA satellites.
  It recently suffered what appears to be a software malfunction (yet to be determined) and different parts of the spacecraft is being restarted in the search for an answer. Each contact lasts less than 15 minutes, in which they must do their trouble shooting, and then it is an agonizing wait until you have the next contact 1-2 hours away (depending on where your next ground station is located).
  As for delay, the longest delay is the one from the control center to the ground station. The RTT from ground to spacecraft is minimal, and uplink bandwidth commonly 2kbps, which is an abundance as you only send small commands. The downlink for telemetry is often 1Mbps (some have 4-32kbps for emergency modes, though). Science downlink may exceed 400Mbps.
  We use geostationary satellite links to reach the site, so network connectivity is not all that bad. The strange part is that commands to a satellite first travels half the globe by ground then out to space over the satellite link, back to earth and then back into space. In about half a second.
  The biggest hurdles of a datacenter in both Arctic and Antarctic regions is the weather conditions (once transport containers weighing several tons where thrown around like LEGO blocks, one was even stacked on top of the others), but also the lack of humidity. Any moving parts containing rubber parts fail rather quickly as the rubber dries out.
  It's all rather fun, though.
6. Re:Space probes by sznupi · 2010-07-23 06:27 · Score: 1
  
  Should be really fun on a mission to Titan - in many ways, worst of both worlds (just, when firmly in the atmosphere, with 100C lower temperature and possibly greater rate of heat loss due to higher density of the atmo...OTOH, that higher density and lower forces involved in buoyancy, might slightly help against convective heat loss? At least the temperatures are quite stable...)
  But nevermind machines, this one will be relatively easy (one our probe survived already, before its batteries got exhausted) - I wonder, do you have anybody at hand there who is quite into survival gear and would find it entertaining to contemplate what would be needed for a human to survive in such environment? :)
  I guess, apart from "scaled up" extreme cold weather outerwear (virtually hermetic & with full helmet?), active heating would be necessary? For example via a garment related to liquid cooling ones in spacesuits, but in this case routing the heat from some carried RTG (hell, even the visor would seem to need quite a lot of it, to avoid condensation in such temp differences). Plus some crafty footwear to isolate from the ground, if only because large part of it is composed of volatiles in "our" temperature ranges...
  (??)
  
  --
  One that hath name thou can not otter
7. Re:Space probes by dargaud · 2010-07-24 23:50 · Score: 1
  
  I wonder, do you have anybody at hand there who is quite into survival gear and would find it entertaining to contemplate what would be needed for a human to survive in such environment? :)
  Well, (raises hand), I have an idea... I'm into mountain climbing as well, so I can combine polar experience, mountain experience and hackery. The longest walk I took at -75C was about 4 hours long (the details are somewhere on my site) . As long as you have a warm place to go back to, you are fine, but for instance we couldn't even drink at those temps, so we came back parched. At -100C a helmet with adequate ventilation to avoid icing would be a good idea. As for clothing and footwear, I don't think heating would be necessary, down suits are quite adequate at -80C and with an extra layer they could go farther down (as long as you don't have to sleep in them). BTW, where do I sign up for Titan ?!?
  
  --
  Non-Linux Penguins ?
You don't need to go as far by mseeger · 2010-07-23 01:16 · Score: 3, Informative

I don't envy someone with a job like that. It is already very difficult to serve locations less remote. E.g. to offer a SLA for a network which spans over 30 locations in the world, we have to make sure that spare parts arrive around the world within a well defined time frame. We don't want anything fancy: a week would be finde by us. But i haven't found any transportation company that guarantees delivery on site in 3rd world countries (big cities) within that time frame. All make exceptions like "customs", which doesn't help me.
CU, Martin
1. Re:You don't need to go as far by i.r.id10t · 2010-07-23 01:20 · Score: 2, Interesting
  
  Or just blow the budget and have a spare set of equipment at each location. When something dies, take out the replacement that is already there, then worry about shipping a new replacement.
  
  --
  Don't blame me, I voted for Kodos
2. Re:You don't need to go as far by pgrb · 2010-07-23 01:25 · Score: 1
  
  On site spares. Make sure you have (tested and shown to be working) spares for everything, and more than one for critical components and ones that are more likely to fail while you are restocking your spares. Engineers are easier to fly around than kit. Yes, it is expensive, and yes, it's a pain to have to store and keep track of the spares inventory (which always seems to 'go walkies'), but if you have a demanding SLA, it's the only way to go.
  
  --
  This line intentionally left..uh..blank?
3. Re:You don't need to go as far by mseeger · 2010-07-23 01:36 · Score: 3, Informative
  
  Or just blow the budget and have a spare set of equipment at each location. When something dies, take out the replacement that is already there, then worry about shipping a new replacement.
  According to the customer, this has been tried and failed... Unused spare equipment has a tendency to vanish.
  CU, Martin
4. Re:You don't need to go as far by mseeger · 2010-07-23 01:41 · Score: 3, Interesting
  
  Yes, it is expensive, and yes, it's a pain to have to store and keep track of the spares inventory (which always seems to 'go walkies'), but if you have a demanding SLA, it's the only way to go.
  Expensive is not the problem, getting the walkies is... According to customer, they experience a strange "honor code" in third world countries... Operative systems seem to be quite safe, but any spare equipment is fair game (Africa is the biggest problem in this regard). Trying to fool the people by faking spare systems to be operative was also not successful.
  CU, Martin
5. Re:You don't need to go as far by nacturation · 2010-07-23 02:26 · Score: 1
  
  According to customer, they experience a strange "honor code" in third world countries... Operative systems seem to be quite safe, but any spare equipment is fair game (Africa is the biggest problem in this regard). Trying to fool the people by faking spare systems to be operative was also not successful.
  I was going to suggest something like this but of course that doesn't solve this particular challenge. How does operational redundancy work? Are live failover/load balanced systems covered by the honor code?
  
  --
  Want to improve your Karma? Instead of "Post Anonymously", try the "Post Humously" option.
6. Re:You don't need to go as far by mseeger · 2010-07-23 02:44 · Score: 1
  
  Are live failover/load balanced systems covered by the honor code?
  It seems to be like this: at a power failure the systems will be turned off. Once the power is restored, they turn the systems one by one. Once everything works, all systems that are still powered off are considered "as spare".
  CU, Martin
7. Re:You don't need to go as far by petermgreen · 2010-07-23 03:09 · Score: 1
  
  Would it be possible to set up a seperate storage location close to where the equipment is used but with much tighter rules on who has access?
  
  --
  note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
8. Re:You don't need to go as far by mseeger · 2010-07-23 03:38 · Score: 1
  
  Has been tried, only increased the amount of damage done.... The idea to ship it was not done lightly, it's last resort...
9. Re:You don't need to go as far by LWATCDR · 2010-07-23 04:24 · Score: 1
  
  I can see the logic of that honor system.
  If you are resourced staved then seeing a spare system just sitting there seems like a huge waste.
  And to them it is.
  Maybe the solution would be to pay the people there such a high wage that value their job. If something walks you fire them.
  
  --
  See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
10. Re:You don't need to go as far by TheOtherChimeraTwin · 2010-07-23 04:43 · Score: 1
  
  Wow. I guess that rules out RAID configurations too (aside from RAID-0).
11. Re:You don't need to go as far by nacturation · 2010-07-23 12:47 · Score: 1
  
  It seems to be like this: at a power failure the systems will be turned off. Once the power is restored, they turn the systems one by one. Once everything works, all systems that are still powered off are considered "as spare".
  What if you ensured that something live is running on each server, even if the server would otherwise do nothing more than standby in the event of a failure? Something like Nagios would tell them whether or not everything is operational and in the event the server went, these "special" services could only be setup remotely by specific highly trained Western engineers. To decoy the decoys, make sure a few cheap-ass spares get setup at the same time without any of these special services and let them go missing. "Oh darn, they took our spare machines, but at least all the core production servers are operational." Meanwhile, sleep peacefully knowing there is redundancy.
  
  --
  Want to improve your Karma? Instead of "Post Anonymously", try the "Post Humously" option.
Gripped? by DogDude · 2010-07-23 01:29 · Score: 0

"If you're ever gripped because you had to haul yourself across town in the middle of the night to fix a server crash, this interview should put things in perspective."

Gripped what? This sentence doesn't make any sense. Are the /. editors even literate?

--
I don't respond to AC's.
1. Re:Gripped? by alphax45 · 2010-07-23 01:34 · Score: 1, Funny
  
  "Are the /. editors even literate?"
  You really have to ask that at this point? :)
  
  --
  K Man
2. Re:Gripped? by FlyingGuy · 2010-07-23 02:11 · Score: 1
  
  Come on dude, it's KDwason what do you expect?
  
  --
  Hey KID! Yeah you, get the fuck off my lawn!
3. Re:Gripped? by tehcyder · 2010-07-23 03:25 · Score: 1
  
  It's gripped, it's sorted, let's off-road. (The Fast Show).
  
  --
  To have a right to do a thing is not at all the same as to be right in doing it
So... by slick7 · 2010-07-23 01:41 · Score: 0

So, is the word "so" so, so?

--
The mind conceives, the body achieves, the spirit manifests.
Griped by Anonymous Coward · 2010-07-23 01:44 · Score: 1, Informative

You know: "I have a gripe with my boss" that sort of thing.
Re:Kdawson . . . by nacturation · 2010-07-23 02:13 · Score: 1

Indeed. "Gripped"? WTH?
Typos on Slashdot? That's nothing to gripe about.

--
Want to improve your Karma? Instead of "Post Anonymously", try the "Post Humously" option.
Amateurs by Anonymous Coward · 2010-07-23 02:26 · Score: 0

Freaking amateurs and their terrestrial data centers.. :)
(j/k great story..)
If you're ever gripped = If you've ever griped? by xxxJonBoyxxx · 2010-07-23 02:35 · Score: 1

If you're ever gripped = If you've ever griped?
Otherwise, I think "gripped" pretty much describes the state of most of the posters here, especially if they get to write about Apple!
Remotest? by Anonymous Coward · 2010-07-23 03:08 · Score: 0

I hate to go grammar nazi here, but seriously, REMOTEST?
I deeply apologize if English is not your first language, otherwise, you are retarded.
1. Re:Remotest? by The+Yuckinator · 2010-07-23 07:17 · Score: 2, Informative
  
  Main Entry: 1remote
  Pronunciation: \ri-mt\
  Function: adjective
  Inflected Form(s): remoter; remotest
  Etymology: Middle English, from Latin remotus, from past participle of removre to remove
  http://east.merriam-webster.com/dictionary/remotest
  (emphasis mine)
2. Re:Remotest? by Anonymous Coward · 2010-07-23 07:41 · Score: 0
  
  I hate to go grammar nazi here, but seriously, REMOTEST?
  I deeply apologize if English is not your first language, otherwise, you are retarded.
  Looks like you need to go back to Grammar school, tough guy.
3. Re:Remotest? by Arker · 2010-07-23 07:55 · Score: 1
  
  To be fair, many of us were taught that one should always say "most remote" instead, and "remotest" sounds a bit bumpkinesque.
  It's still much better than the third alternative, "most remotest," of course.
  
  --
  =-=-=-=-=-=-=-=-=-=-=-=-=-=-
  Friends don't let friends enable ecmascript.
Lets Just hope they arent running Windows Servers by Anonymous Coward · 2010-07-23 03:21 · Score: 0

Hopefully they aren't running Windows, with all the system related reboots that are needed every few days.

Gods I hate Tuesdays, waiting on patches, second only to Wednesdays fixing everything the patch broke.
Keep it cool? by Snaller · 2010-07-23 06:17 · Score: 1

Just open a window!

--
If Google really cared they would fix Android Chrome to reflow text, instead of discriminating
Forget global warming by myshadows · 2010-07-23 07:30 · Score: 1

I think just this data center and the observatory is enough to bring down the south pole ice caps... If this is taking place at the south pole, shouldn't this be concerning? Also, most of the IT staff can manage the center remotely with a minimum number of people required at base camp. With regards to power, I believe geothermal is a possibility but I don't know how that would interfere with their observations.
Redundancy Redundancy Redundancy by nurb432 · 2010-07-24 04:47 · Score: 1

Couple that with some good planning and it shouldn't be a big deal to remotely manage it for months on end. Once a year you haul out upgrades and replacements.
Virtualization is your friend.

--
---- Booth was a patriot ----