NASA Loses Contact With Space Station Over Software Update

← Back to Stories (view on slashdot.org)

NASA Loses Contact With Space Station Over Software Update

Posted by Soulskill on Tuesday February 19, 2013 @08:08AM from the i-didn't-know-they-used-itunes dept.

kodiaktau writes "Reports early Tuesday morning say that a software update to the International Space Station caused a communications blackout with Houston control. Remediation of the update has allowed the astronauts limited communication every 90 minutes or so. It is expected that the issue will be resolved today."

36 of 111 comments (clear)

Min score:

Reason:

Sort:

Should've read the manuals. by ZaMoose · 2013-02-19 08:10 · Score: 5, Funny

I mean, even my Linksys warns me to only update firmware when I've got an Ethernet cable plugged in to it, because you know how wireless upgrades go.

--
I wish I had a kryptonite cross, because then you could keep Dracula and Superman away.
1. Re:Should've read the manuals. by eksith · 2013-02-19 08:12 · Score: 4, Funny
  
  They don't make CAT6a cables that long :/
  
  --
  If computers were people, I'd be a misanthrope.
2. Re:Should've read the manuals. by tehcyder · 2013-02-19 22:26 · Score: 2
  
  Also, we don't care. Many of us software developers make more than $65/hour "on the computer."
  But software developers don't generally get paid just for masturbating in front of total strangers over the internet. It's more of a hobby.
  
  --
  To have a right to do a thing is not at all the same as to be right in doing it
What could possibly go wrong? by IronHalik · 2013-02-19 08:11 · Score: 5, Informative

http://imgur.com/gallery/fv2rg9a
1. Re:What could possibly go wrong? by MrEricSir · 2013-02-19 08:27 · Score: 4, Funny
  
  And his next Tweet: https://twitter.com/Cmdr_Hadfield/status/303807080433860608
  Scary!
  
  --
  There's no -1 for "I don't get it."
2. Re:What could possibly go wrong? by Ol+Biscuitbarrel · 2013-02-19 09:03 · Score: 2
  
  I'm so freaked out I had to take a stress pill and think things over.
The rumor has it... by toxygen01 · 2013-02-19 08:15 · Score: 5, Funny

that it was Java update... what to say...
Windows 8 by puddingebola · 2013-02-19 08:16 · Score: 5, Funny

Actually, it was just some confusion when someone right swiped on Windows 8. The Charms bar came up and then the weather app launched and nobody knew how to close either one.
1. Re:Windows 8 by RocketRabbit · 2013-02-19 08:48 · Score: 2
  
  The mere fact that it is called a "charms bar" just makes me squirm in disgust. What the fuck were they thinking?
2. Re:Windows 8 by Dunbal · 2013-02-19 08:50 · Score: 3, Informative
  
  They were thinking "How can we grab some of Apple's share?" and apparently a "charms bar" is the way to go.
  
  --
  Seven puppies were harmed during the making of this post.
3. Re:Windows 8 by Megane · 2013-02-19 09:08 · Score: 5, Funny
  
  They had to do something to compete with the Genius Bar at Apple stores!
  I hear Ubuntu is going to introduce a Granola Bar to compete with both of them.
  
  --
  #naabhaprzrag, #sverubfr-000, #agi-fcbafberq, negvpyr[pynff*=' negvpyr-ary-'] { qvfcynl: abar !vzcbegnag; }
4. Re:Windows 8 by steelfood · 2013-02-19 09:24 · Score: 3, Funny
  
  The Charms bar came up
  I'll bet the problem arose because they thought it was strange.
  
  --
  "If a nation expects to be ignorant and free in a state of civilization, it expects what never was and never will be."
Communication Reestablished by NoKaOi · 2013-02-19 08:19 · Score: 5, Informative

http://www.space.com/19853-space-station-contact-lost-nasa.html

Update: NASA has reestablished contact with the International Space Station. For the latest news, read: NASA Restores Contact with Space Station

As far as NASA officials can tell, the space station's loss of communications was unrelated to the software update, Kelly Humphries, a public affairs specialist at NASA told SPACE.com. It was a coincidence that the space agency lost contact with the station as the computers were being updated.
Yep by roc97007 · 2013-02-19 08:20 · Score: 4, Insightful

Every time the offshore admins want to apply an update, I ask them "what is your contingency plan should you brick the server?" and they always answer "Call the vendor". Sigh.

--
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
1. Re:Yep by omglolbah · 2013-02-19 08:30 · Score: 5, Interesting
  
  We do a full image backup of the server.
  Then we shut it down (they're all redundant) and remove one set of drives from the mirrored raid.
  Start back up.
  Run the update.
  Verify that the update went ok
  Perform new image backups.
  When everyone is satisfied shove the mirrored drives back in.
  Then again, we're "offshore" as in an oil rig and patching control system HMI servers... so I guess having a contingency plan would be required. This rig (where I am at now :p) makes 50 million USD a day in natural gas.. so uptime is paramount!
2. Re:Yep by roc97007 · 2013-02-19 08:41 · Score: 4, Insightful
  
  That's a different, and probably more appropriate definition of "offshore". That also sounds like a very good procedure. I would have added "we only upgrade firmware to solve specific problems and vulnerabilities appropriate to our environment, not just because it's 'the latest'". But I'm told I'm too conservative.
  Here, the admins are offshore (as in, physically on the other side of the world) but the machines are still local. They've rebadged former mainframe operators to be "hands and eyes" in case a button has to be pushed or a memory stick changed out. Of course, "hands and eyes" have had no hardware training whatsoever. I made some fuss recently when I caught one of them changing out a memory stick with the gator clip on their wrist strap dangling in the air. The response was to raise the issue as to why I still had access to the computer room? Geh.
  
  --
  Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
3. Re:Yep by war4peace · 2013-02-19 08:46 · Score: 2
  
  Offtopic, sort of, but never have I seen, in about 8 years or so, a computer component being toasted by static discharge, and here nobody ever uses grounding when they work with hardware.
  
  --
  ...gis sdrawkcab (usually not responding to ACs; don't bother posting as AC)
4. Re:Yep by AmiMoJo · 2013-02-19 09:18 · Score: 2
  
  I knew an admin who was so scared of updates breaking something and getting called up to London at 6 PM to fix it he just never bothered. Every time the staff infected the RDP server with a virus because they were stuck with IE7 he blamed them, and one even lost her job over it.
  Eventually the company got fed up, ditched us and found another support company. The first thing they did was install all the updates and virtualize all the servers.
  
  --
  const int one = 65536; (Silvermoon, Texture.cs)
  SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
5. Re:Yep by asc99c · 2013-02-19 09:18 · Score: 2
  
  Neither have I, and I don't bother with any precautions when working on PCs. But I have also assembled an AIX server which would have been just about into six figures in US dollars, and thought for that one I'd spend five minutes to go and find the anti-static equipment!
6. Re:Yep by roc97007 · 2013-02-19 12:15 · Score: 2
  
  Large app servers, database back-ends, and outward facing servers remain physical. Just about everything else is a candidate for virtualization. It so happens that my responsibilities are primarily with big physical servers.
  But let's talk virtualization... you haven't seen a mess until you've seen a firmware update pushed out that takes three-quarters of your VM farm offline. Virtualization allows you to be responsible, by migrating off a few servers at a time, upgrading them, and if successful, move the instances back. But if anything a big VM farm gives the irresponsible an even greater opportunity to shoot one's self in the foot. Since most the hardware is identical, why not patch them all at once and save time? And then... oops.
  
  --
  Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
Hello, IT by Clancie · 2013-02-19 08:21 · Score: 5, Funny

Have you tried turning it off and on again?
1. Re:Hello, IT by NoNonAlphaCharsHere · 2013-02-19 08:57 · Score: 5, Funny
  
  Also, being a space station, it explains why you didn't hear the 'WHOOSH'.
2. Re:Hello, IT by steelfood · 2013-02-19 09:26 · Score: 2
  
  Before that, check to see if the power cord's still plugged in.
  
  --
  "If a nation expects to be ignorant and free in a state of civilization, it expects what never was and never will be."
I don't think they update on the fly they send the by Joe_Dragon · 2013-02-19 08:23 · Score: 2

I don't think they update on the fly they send the files and verify that they are 100% before starting a install locally
Don't worry, it's not bricked. by Nanoda · 2013-02-19 08:25 · Score: 5, Funny

Someone is going to have to hold down the button on the side for 10 seconds though.
1. Re:Don't worry, it's not bricked. by UltraZelda64 · 2013-02-19 08:46 · Score: 3, Funny
  
  Smoke on the water... a massive brick in the sky...
Screen is stuck on this: by Bill+Hayden · 2013-02-19 08:32 · Score: 3, Funny

Applying update 8 of 27...

Please do not turn off your computer.

--
Protect your browser with the Force Safe Search add-on
No one ever goes to the source by Anonymous Coward · 2013-02-19 08:35 · Score: 2, Insightful

This has been fixed for over three hours.
http://www.nasa.gov/mission_pages/station/expeditions/expedition34/e34_021913.html
Is it really that hard to check with NASA?
1. Re:No one ever goes to the source by PRMan · 2013-02-19 09:38 · Score: 2
  
  Are you kidding? I'm just happy when Slashdot isn't 3 DAYS behind. 3 hours is a massive improvement.
  
  --
  Peter predicted that you would "deliberately forget" creation 2000 years ago...
Re:Redundant systems? by sv_libertarian · 2013-02-19 08:37 · Score: 2

They also have an amateur radio station on board that can be used for emergency communications. It would have been a CB, but West Virginia couldn't get their mobile home module into orbit.
Re:I don't think they update on the fly they send by war4peace · 2013-02-19 08:46 · Score: 3, Insightful

Yes. Have you heard of grammar?

--
...gis sdrawkcab (usually not responding to ACs; don't bother posting as AC)
Re:Redundant systems? by A10Mechanic · 2013-02-19 09:03 · Score: 2

As an amatuer radio operator, I'd love to take up the slack and do comms for them. Get enough of us, and we can provide global support, free of charge, and would be honored to do it! There *may* be some coverage holes in the south Pacific...
Doesn't matter... by mschaffer · 2013-02-19 09:22 · Score: 2

It doesn't matter as it's further than 100 meters away.
1. Re:Doesn't matter... by tlhIngan · 2013-02-19 10:28 · Score: 2
  
  It doesn't matter as it's further than 100 meters away.
  That limit is because of how long it takes for a minimum sized packet to traverse the network for collision detection. If you can guarantee there won't be collisions (a switch does this) then you're pretty much not limited by length anymore other than signal strength. All you need is a bunch of switches to boost the signal up and you're golden.
  Back in ye olde days of hubs, they were half duplex because the transmitting NIC would send bits down the line and read the bits off the received end. If the bits didn't match, it was a collision and the detecting card jammed the link to tell everyone to back off. As the minimum packet was 64 bytes, the length limit came into play to allow the packet to reach the other end before transmitting stopped so if it sent the packet with no collisions, it was reasonably confident the packet made it intact. If it was too long, the NIC could send the packet out and another host could transmit, causing the packet to be corrupted, and the host wouldn't see that its packet got corrupted.
  In this day and age of full duplex and switches everywhere, it's less of an issue.
Re:News for Nerds by DickBreath · 2013-02-19 10:49 · Score: 2

> Sadly, I get better news from CNN.com.

Okay, see this: CNN Anchor Asks Bill Nye If Global Warming Had Anything To Do With A Near-Earth Asteroid.

--

I'll see your senator, and I'll raise you two judges.
Software Verification and Validation by DERoss · 2013-02-19 10:53 · Score: 4, Interesting

I spent over 20 years of my career (now retired) working for a company that did independent verification and validation (IV&V) of software used by the military to operate its unmanned space satellites. Not once was a satellite lost from an error in the software if we were involved.
There were some 10 or more other, unrelated companies developing software for various space satellites. We did more than merely test the resulting products. We started by reviewing the developers' design documents; our reviews required responses or revisions before any coding could occur. Next we reviewed the developers' programming documents; our reviews required responses or revisions before programming could be completed. Then we reviewed the developers' test documents; our reviews required responses or revisions before the developers could conduct their own internal unit tests. We attended the conduct of those internal tests and audited the results to ensure that the purposes and criteria of the tests were satisfied.
Finally, the developers would deliver their software to us. We would test the products at the package and system level. We looked at how products from different developers interfaced with each other, whether human interfaces were reasonable, and whether the government's requirements had been met. Our test documents were reviewed by the military organizations that would be using the software, and we did not start testing until we responded or revised our test documents.
This IV&V process approximately doubled the cost of providing software. However, no such software caused a satellite to land on the White House or (worse) on the Kremlin. In the early 1990s, the Pentagon decided to save money by eliminating IV&V. I continued testing software for military satellites, but then it was within the companies that developed the software. When schedules or costs were at risk, testing was cut short.
Sic transit gloria mundi.