Why ISS Computers Failed

Geoffrey.landis writes "It was only a small news item four months ago: all three of the Russian computers that control the International Space Station failed shortly after the Space Shuttle brought up a new solar array. But why did they fail? James Oberg, writing in IEEE Spectrum, details the detective work that led to a diagnosis." The article has good insights into the role the ISS plays as a laboratory for US-Russian technology cooperation — something that is likely to be crucial in any manned Mars mission.

Comments

The REAL reason they failed

[Rebelgecko]

They "upgraded" to Vista.

Re:The REAL reason they failed

[Anonymous Coward]

Clippy: It looks like you want to install a new solar array. Do you want help with that?

They didn't bring the right travel adapters.

[NotQuiteReal]

Metric electricity vs Imperial electricity...

Re:They didn't bring the right travel adapters.

[Cassius Corodes]

Proletarian electricity refused to mix with bourgeoisie electricity.

Urgh.

[Airconditioning]

The article reeked of condesension towards the Russians. It's no way to report on your partners in space.

Re:Urgh.

[Jugalator]

I agree... That's what first came to mind after having watched this incident unfold live. What he fails to mention is that the Russian engineers were always open to suggestions and they cooperated pretty well when they needed to discuss the problems. The Russians were also working nearly 24/7 on trying to find and resolve the problems and come up with theories before they were running out of time. The article makes it sound like they early on got locked into "blaming the Americans" or something. It was merely one theory that was tossed around and discussed, and diagnosed early on. If there seem to be a power failure (which it ended up being all about), surely one logically suspected culprit could be a power feed problem?

Re:Urgh.

[Ethanol-fueled]

Hell yeah. Mod parent up. The real heroes are in space cooperating and solving problems.
Seriously, all of that political cold war-era cockwaving should stop.

Duct tape saves the day!

[Cyberax]

...They also decided to rig a thermal barrier out of a surplus reference book and all-purpose gray tape....

Once again, duct tape saves the day! :)

Re:Duct tape saves the day!

[p00n0s]

A person needs only three tools in life: WD-40, duct tape and a hammer. If it doesn't move and it should, use the WD-40. If it moves and it shouldn't, use the duct tape. If either doesn't work, use the hammer.

Hmmm

[K.os023]

Could this be the one place where it would be appropriate to mention that in Russia, crashes compute?


Or would that be "In Russia, crashes compute you!" ?

Redundancy != Safety

[quanticle]

I think NASA should have learned this lesson by now. After all, the Challenger disaster showed this principle as well. In that case, the same cold temperature that weakened the primary seal on the solid rocket booster weakened the secondary as well, sapping its ability to provide redundant backup. In this case, the same condensation affected all three computers equally.

Its troubling to see them taking shortcuts on safety and redundancy, when such measures have resulted in loss of life before. How hard would it have been to have had three shut-off cables?

Re:Redundancy != Safety

[8-bitDesigner]

Two nit-picky points here:

1. It wasn't condensation that felled all three computers, it was a single corroded connector, which shorted and sent a kill-command to all three computers. Technically, redundancy here would've circumvented that issue.
2. Actually, I believe the article stated that it was a Russian-manufactured component, not a NASA design.

Proper debugging technique

[dd1968]

These computers functioned for months or years. When they failed, the right question to ask first was "what has changed?" This is exactly what the Russians did. According to the author the Russians first considered potential causes stemming from the newly installed solar power wing, the visiting shuttle, and the expanded station structure (the reason for the shuttle being there). One conclusion is that they were pointing the finger at NASA and playing the blame game. Another is that they were doing what good engineers anywhere would do to debug the problem.

The author is obviously way more qualified than I to assess the situation and he may well be right but from the content of the article I came away thinking, wow, I would have looked first at all the recent changes to the station and the power supply too.

Hmmmm.

[WindBourne]

The original plans called for the ISS to be finished many years ago. It is not yet, because America has had issues with transportation. In addition, a few modules that were planned to make the ISS very useful were canceled because of us (in particular, CAM). In the end, both sides have had issues, and changes have occurred. That is normal for these kinds of projects. To be honest, I think that all of this has been handled pretty decently.

Power off command

[jsse]

Also, in a shocking design flaw, there was a "power off" command leading to all three of the supposedly redundant processing units. That reminds me many years ago, when my friend worked as a programmer in a major bank writing small programs for an online international financial system. He issued an 'shutdown' command through JCL(Job Control Language) and that really shutdown the entire system. He didn't realize he had the privilege to issue administration commands. Instead of reporting the crisis to his manager, he hide away until someone figured out what's going on. Needless to say, my friend was fired.

Years later I met his manager, he told me that my friend could have been promoted for discovering one of the biggest loophole ever in the bank's history, if he had reported the problem immediately. Though the unexpected shutdown caused considerable damage, it could have saved billions from real break-in with this loophole.

That's a lesson that every engineer should have been learned. :)

Here we go again...

[LanceUppercut]

Well, well, well... Here we go again. Jim Oberg. That same Jim Oberg who was almost blowing his gasket a couple of weeks ago when that journalist was asking him questions about alcohol abuse by astronauts (you all remember the story, I'm sure). It was all preposterous nonsense not backed up by any evidence, he said, berely keeping his cool. And what do we see now? He is happily making up stories about Russians accusing US of the computer falures - something that never happened in reality. The power problems caused by some new US installations were indeed considered as intermediate working brainstormed versions of what could have happened. But nobody ever did any fingerpointing or made any acussations before the situation was sufficiently researched and the root cause determined. Of course, Jim Oberg could not refreain from distorting the truth "just a little". Tsk, tsk, tsk... Note, how he refers to the hypothesis as both "blatant finger pointing" and just "guesses" within single paragraph - just to keep his article a little fuzzy, so that he can flip-flop to either when the situation calls for it. Nothing surprising here, though...

The computers are not Russian, but European

[hazard]

The article is misleading. The computers are not actually of Russian make, they were supplied to Russians by Europeans (EADS). See here.

Indeed, how many russion casualties have there bee

[SmallFurryCreature]

Tell me, how many casualties have the russians had in the last decade, even last two decades? This was in the days of Mir, when the russians maintained a continues space pressence year after year and the US was out of space for year after year for blowing up space shuttles.

So whose tech is behind whose? The ISS didn't plunge out of the sky when the Space Shuttle was not available, apparently the russian capability is more then enough to operate it.

And finally, who build the de-humidefier that was the fault in the first place?