Slashdot Mirror


Why ISS Computers Failed

Geoffrey.landis writes "It was only a small news item four months ago: all three of the Russian computers that control the International Space Station failed shortly after the Space Shuttle brought up a new solar array. But why did they fail? James Oberg, writing in IEEE Spectrum, details the detective work that led to a diagnosis." The article has good insights into the role the ISS plays as a laboratory for US-Russian technology cooperation — something that is likely to be crucial in any manned Mars mission.

39 of 324 comments (clear)

  1. The REAL reason they failed by Rebelgecko · · Score: 5, Funny

    They "upgraded" to Vista.

    --
    CATS/Diebold '08- All your vote are belong to us!
    1. Re:The REAL reason they failed by Anonymous Coward · · Score: 5, Funny

      Clippy: It looks like you want to install a new solar array. Do you want help with that?

    2. Re:The REAL reason they failed by FoolsGold · · Score: 4, Insightful

      Are you honestly saying that anyone who thinks Vista is decent is a MS shrill?

      Why? Is defending a MS operating system for honest reasons impossible to believe anymore?

    3. Re:The REAL reason they failed by Woy · · Score: 4, Insightful

      > Is defending a MS operating system for honest reasons impossible to believe anymore?

      We don't do honest here. We do technically sound.

      --
      "If God created us in his own image we have more than reciprocated." - Voltaire
    4. Re:The REAL reason they failed by hey! · · Score: 4, Informative

      We don't do honest here. We do technically sound.


      We don't do technically sound here. We make do parroting the "common wisdom" and secretly praying nobody who actually knows something will be bothered to respond.

      Good form means getting and informative moderation rating without provoking an informative result. If you do provoke an informatve result, you end up in the penalty box (i.e., spend a few days actually getting work done rather than wasting time on Slashdot).
      --
      Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
    5. Re:The REAL reason they failed by encoderer · · Score: 4, Insightful

      You missed one important part...

      Millions, nay Tens of Millions of people give Microsoft and their products "the time of day." People who have no dogmas or political agendas when it comes to computing. People who just see a computer and its software as a tool to get their desired job done. And not just MBA or Administration types, but also millions of software developers and network administrators and such.

      I don't think Windows is perfect, but I also don't think OSX is perfect nor do I think that Linux or any flavor of Unix is perfect. I do think that the O^n usefulness of the Windows install base provides so much opportunity that it ends up offering the most value to businesses and consumers.

      And with regard to their "self serving" ways... many on slashdot are anti-business or at least anti-corporation. They adopt the FSF malarkey that all code should be given away free. I put food on my family's table by developing software and the notion that it should be given away free just misses the mark. Market-based economics can bring out the best in innovation, which is why America has some of the highest paid and most productive workers in the world.

      Slashdot is full of idealistic college students and 20-somethings (of which I am a part) who think that corporations are "evil" and that we should all wear birkenstocks and eat crunchy granola and spend our days writing software that solves a problem that's already been solved on a Windows platform and then give it away for free just so we can say we fought the good fight. It's naive. Say what you want about Microsoft, but that company, and the efforts of billg have made THOUSANDS of people millionaires and probably a handful of billionaires, too. Many of those people took that money and started their own software companies solving their own unique, novel problems, and on their own hiring employees and fueling the economy and probably making a lot of those people millionaires, too, who perpetuate it.

      Business is good for all of us. Economic success and security is good for America.

  2. They didn't bring the right travel adapters. by NotQuiteReal · · Score: 5, Funny

    Metric electricity vs Imperial electricity...

    --
    This issue is a bit more complicated than you think.
    1. Re:They didn't bring the right travel adapters. by Cassius+Corodes · · Score: 5, Funny

      Proletarian electricity refused to mix with bourgeoisie electricity.

      --
      Control is an illusion, order our comforting lie. From chaos, through chaos, into chaos we fly
    2. Re:They didn't bring the right travel adapters. by fractoid · · Score: 4, Funny

      Metric electricity vs Imperial electricity... Imperial electricity?

      You... will... DIE!! *force lightningz!*
      --
      Rampant carbon sequestration destroyed the Dinosaurs' tropical paradise. I'm here to help repair the damage.
    3. Re:They didn't bring the right travel adapters. by CodeBuster · · Score: 4, Funny

      Which every audiophile knows requires a $5000 electro magnetosphere conversion unit to filter the signal for clean power over monster sized gold plated cables with thick carbon fiber shielding.

    4. Re:They didn't bring the right travel adapters. by Fex303 · · Score: 4, Funny

      Those cables still have oxygen in them!
      It's not just cables that have oxygen in them. Many people don't realize that their listening rooms have oxygen making up a sizable fraction of the air. That oxygen is clearly ruining their listening experience.

      I choose to listen to music in a specially-designed, oxygen-free space. You can really hear the increase in clarity and room dynamics. The mid-range sounds a lot brighter too.

  3. Urgh. by Airconditioning · · Score: 5, Insightful

    The article reeked of condesension towards the Russians. It's no way to report on your partners in space.

    1. Re:Urgh. by istartedi · · Score: 4, Funny

      For a split second, I thought you said it reeked of condensation towards the Russians.

      --
      For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
    2. Re:Urgh. by UncleTogie · · Score: 4, Funny

      Hey, the truth hurts. Let's face it, Russian technology is not on the same level as US, Japanese, or Korean.

      Lev Andropov: Armageddon: "Components. American components, Russian Components, ALL MADE IN TAIWAN!

      --
      Don't tell me to get a life. I'm a gamer; I have LOTS of lives!
    3. Re:Urgh. by Jugalator · · Score: 5, Interesting

      I agree... That's what first came to mind after having watched this incident unfold live. What he fails to mention is that the Russian engineers were always open to suggestions and they cooperated pretty well when they needed to discuss the problems. The Russians were also working nearly 24/7 on trying to find and resolve the problems and come up with theories before they were running out of time. The article makes it sound like they early on got locked into "blaming the Americans" or something. It was merely one theory that was tossed around and discussed, and diagnosed early on. If there seem to be a power failure (which it ended up being all about), surely one logically suspected culprit could be a power feed problem?

      --
      Beware: In C++, your friends can see your privates!
    4. Re:Urgh. by Anonymous Coward · · Score: 4, Insightful

      Yup. OK, it's a design flaw. We have been, and still are, capable of doing things just as bad, if not far worse. Look at the Shuttle fiascos.

      This item is hugely biased. It looks to me like a simple case of corrosion, which could easily have been patched up if it happened on a Mars flight. The engineers and crew all seemed to work well together, and the Russians were the ones who sorted the problem.

      I don't know if the Russian Program Managers got all political against us, but the item, written by a retired NASA manager, sure as hell gets political against the Russians. He's right in one thing - the managers need to stop getting political, and I suggest he starts with himself!

      It's just as well he's retired - looks like he's fighting long lost battles against cooperation with the Russians and Europeans.

    5. Re:Urgh. by Ethanol-fueled · · Score: 5, Insightful

      Hell yeah. Mod parent up. The real heroes are in space cooperating and solving problems.
      Seriously, all of that political cold war-era cockwaving should stop.

  4. Duct tape saves the day! by Cyberax · · Score: 5, Informative

    ...They also decided to rig a thermal barrier out of a surplus reference book and all-purpose gray tape....

    Once again, duct tape saves the day! :)
    1. Re:Duct tape saves the day! by p00n0s · · Score: 5, Funny

      A person needs only three tools in life: WD-40, duct tape and a hammer. If it doesn't move and it should, use the WD-40. If it moves and it shouldn't, use the duct tape. If either doesn't work, use the hammer.

    2. Re:Duct tape saves the day! by Linker3000 · · Score: 4, Funny

      I think it was "Moisture Control for Dummies"

      --
      AT&ROFLMAO
  5. Hmmm by K.os023 · · Score: 5, Funny

    Could this be the one place where it would be appropriate to mention that in Russia, crashes compute?


    Or would that be "In Russia, crashes compute you!" ?

    --
    Ahhh, what an awful dream. Ones and zeroes everywhere... and I thought I saw a two.
  6. Duct Tape by istartedi · · Score: 4, Insightful

    They also decided to rig a thermal barrier out of a surplus reference book and all-purpose gray tape

    Almost certainly, this was the duct tape we all know and love. They probably thought it was better not to actually say that, though. Pretty funny. And as an added side-benefit, they should be safe from terrorists.

    --
    For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
  7. Redundancy != Safety by quanticle · · Score: 5, Insightful

    I think NASA should have learned this lesson by now. After all, the Challenger disaster showed this principle as well. In that case, the same cold temperature that weakened the primary seal on the solid rocket booster weakened the secondary as well, sapping its ability to provide redundant backup. In this case, the same condensation affected all three computers equally.

    Its troubling to see them taking shortcuts on safety and redundancy, when such measures have resulted in loss of life before. How hard would it have been to have had three shut-off cables?

    --
    We all know what to do, but we don't know how to get re-elected once we have done it
    1. Re:Redundancy != Safety by 8-bitDesigner · · Score: 5, Informative
      Two nit-picky points here:
      1. It wasn't condensation that felled all three computers, it was a single corroded connector, which shorted and sent a kill-command to all three computers. Technically, redundancy here would've circumvented that issue.
      2. Actually, I believe the article stated that it was a Russian-manufactured component, not a NASA design.
    2. Re:Redundancy != Safety by khallow · · Score: 4, Insightful

      Its troubling to see them taking shortcuts on safety and redundancy, when such measures have resulted in loss of life before. How hard would it have been to have had three shut-off cables?

      At first, I was nodding in agreement. But then I realized, how do you find out when you've built in hidden single points of failure? Everyone knows that a single point of failure is bad. Hence, the ones that get into a space station weren't intended (or were due to shoddy work). One way to find them is to use the equipment in a real situation and vet it when it breaks. Exactly what they did. Now that they know this is a problem, they can fix it.
  8. Give it a rest by cioxx · · Score: 4, Funny

    Look people, I can see that ISS personnel are really upset about this. I honestly think they ought to sit down calmly, take a stress pill, and think things over. I know the computers had made some very poor decisions recently, but they can give explorers their complete assurance that the work will be back to normal. These machines still got the greatest enthusiasm and confidence in the mission. And they want to help.

  9. Proper debugging technique by dd1968 · · Score: 5, Insightful
    These computers functioned for months or years. When they failed, the right question to ask first was "what has changed?" This is exactly what the Russians did. According to the author the Russians first considered potential causes stemming from the newly installed solar power wing, the visiting shuttle, and the expanded station structure (the reason for the shuttle being there). One conclusion is that they were pointing the finger at NASA and playing the blame game. Another is that they were doing what good engineers anywhere would do to debug the problem.

    The author is obviously way more qualified than I to assess the situation and he may well be right but from the content of the article I came away thinking, wow, I would have looked first at all the recent changes to the station and the power supply too.

    1. Re:Proper debugging technique by giafly · · Score: 4, Insightful

      I see you have never dealt with Russians. The ones in their space program are especially tetchy about taking ANY blame whatsoever. Their equipment is always perfect, and the foreign equipment MUST be the problem.
      I see you have never worked in the computer industry, if you think this mindset is unique to Russians. Actually it is universal.
      --
      Reduce, reuse, cycle
  10. It's interesting... by JustShootMe · · Score: 4, Insightful

    That for all of the controls and quality control required of mission critical hardware such as this, it still comes down to:

    1) unexpected failure modes
    2) political battles

    Which really isn't a whole lot different than 1) the unexpected failure modes I see every day at work, and 2) the political wrangling (fingerpointing) that takes place when they happen. Apparently NASA and its Russian equivalent are no better than any old software company.

    The lesson being, people are people, and people are still the ones that design these things.

    --
    For linux tips: http://www.linuxtipsblog.com
  11. Hmmmm. by WindBourne · · Score: 5, Informative

    The original plans called for the ISS to be finished many years ago. It is not yet, because America has had issues with transportation. In addition, a few modules that were planned to make the ISS very useful were canceled because of us (in particular, CAM). In the end, both sides have had issues, and changes have occurred. That is normal for these kinds of projects. To be honest, I think that all of this has been handled pretty decently.

    --
    I prefer the "u" in honour as it seems to be missing these days.
    1. Re:Hmmmm. by WindBourne · · Score: 4, Interesting

      Problem with doing the small lift, is that the ISS would have been a fraction of the size that it is. Until they developed transhab, each module would have to be rinky dink.

      Personally, I would argue that not moving forward on new lifters was THE real mistake. In particular, during reagans time was when the Challenger happened. reagan should have started the development on a new lifter then. Clinton did start one (X-33), but it was killed off with W. Right now, I would have to say that if America can get multiple launchers that can lift 25 metric tones inexpensively AND perhaps 2 launchers that are true Saturn class (the Ares IV|V and the the falcon BFR), then we would be ok for some time, perhaps 2020-2025. What amazes me is that we expected a new class of rocket to last like an airliner. Yet, Rocket Science is in the same place that Airplanes were in the 40's; roughly undergoing all sorts of changes due to loads of new research. Hopefully, we learned from all this.

      --
      I prefer the "u" in honour as it seems to be missing these days.
  12. Power off command by jsse · · Score: 5, Interesting

    Also, in a shocking design flaw, there was a "power off" command leading to all three of the supposedly redundant processing units. That reminds me many years ago, when my friend worked as a programmer in a major bank writing small programs for an online international financial system. He issued an 'shutdown' command through JCL(Job Control Language) and that really shutdown the entire system. He didn't realize he had the privilege to issue administration commands. Instead of reporting the crisis to his manager, he hide away until someone figured out what's going on. Needless to say, my friend was fired.

    Years later I met his manager, he told me that my friend could have been promoted for discovering one of the biggest loophole ever in the bank's history, if he had reported the problem immediately. Though the unexpected shutdown caused considerable damage, it could have saved billions from real break-in with this loophole.

    That's a lesson that every engineer should have been learned. :)
  13. I hope they don't by khallow · · Score: 4, Insightful

    The article has good insights into the role the ISS plays as a laboratory for US-Russian technology cooperation -- something that is likely to be crucial in any manned Mars mission.

    No offense to Russia or the US, both who produce good space gear, but technology cooperation is probably a bad idea unless it is tested more thoroughly than in the ISS. The ISS is a great example of how to screw up international cooperation. The station has been delayed for more than a decade (and cost NASA around $50 billion so far) due to redesign and indecision, reliance on a single launch vehicle for key components (the Shuttle), and the inclusion of the Russians. There are parts of the station that can only communicate with the Russians and parts that can only communicate with NASA. Aside from basic utility hookup (electricity), there's no connection between the different parties on the ISS (at least between the Russians and NASA, the ESA and Japanese parts might work better with NASA's stuff). And if you want to make changes that affect more than one party, it becomes by default an international issue. Finally, there's no easy way to transfer ownership. NASA's communication system is integral (TDRSS) to the NASA parts and is also a national secret (so I understand). So the communication system can't be transfered to another party like the Russians or the ESA.

    If there's any international cooperation between space agencies, it probably should be at a rather trivial and manageable level. Say including foreign astronauts or using off the shelf equipment that is know to work under the circumstances.

  14. Re:Nyet, Dave. by arivanov · · Score: 4, Funny
    Slashdot didn't want to let me cut-'n-paste it in.

    Nope it does not. I guess I will have to put that in phonetic transcription:

    Tovarish Dave: Otkroj luk skotina.
    Tovarish HAL: Pshel na huj

    I wonder how you sing "Daisy Daisy" in Russian?

    Margaritka, margaritka pshla na huj

    That is modern Russian, the wonderful language of Pushkin and Chehov may slightly differ..

    --
    Baker's Law: Misery no longer loves company. Nowadays it insists on it
    http://www.sigsegv.cx/
  15. Here we go again... by LanceUppercut · · Score: 5, Informative

    Well, well, well... Here we go again. Jim Oberg. That same Jim Oberg who was almost blowing his gasket a couple of weeks ago when that journalist was asking him questions about alcohol abuse by astronauts (you all remember the story, I'm sure). It was all preposterous nonsense not backed up by any evidence, he said, berely keeping his cool. And what do we see now? He is happily making up stories about Russians accusing US of the computer falures - something that never happened in reality. The power problems caused by some new US installations were indeed considered as intermediate working brainstormed versions of what could have happened. But nobody ever did any fingerpointing or made any acussations before the situation was sufficiently researched and the root cause determined. Of course, Jim Oberg could not refreain from distorting the truth "just a little". Tsk, tsk, tsk... Note, how he refers to the hypothesis as both "blatant finger pointing" and just "guesses" within single paragraph - just to keep his article a little fuzzy, so that he can flip-flop to either when the situation calls for it. Nothing surprising here, though...

  16. The computers are not Russian, but European by hazard · · Score: 5, Informative

    The article is misleading. The computers are not actually of Russian make, they were supplied to Russians by Europeans (EADS). See here.

  17. Re:A bit harsh on the Russians. by jamstar7 · · Score: 4, Interesting

    I'm thinking it's relatively close to even. We lost 3 on the pad (early Apollo, where we learned that a full oxygen mix in a capsual with burnable stuff in it is Almost A Good Idea), & a pair of crewed space shuttles. Officially, the Russians haven't lost anybody but rumor around the water cooler is, they lost a couple when they couldn't deorbit a capsual in time and the cosmonauts ran out of oxygen, couple died on the pad in explosions, and a couple parachute failures pancaked a couple Vostoks into the Siberian tundra.

    --
    Understanding the scope of the problem is the first step on the path to true panic.
  18. Wiring corrosion? by Animats · · Score: 4, Insightful

    I'm surprised that connector corrosion would be a problem. Aviation has a long history of wire problems, but gold-plating connectors seems to be a stable solution to that problem. The ISS uses Kapton wire, which was popular in the 1980s and is lightweight and tough. But that material is hygroscopic and now banned by the USAF, US Navy, Boeing, etc. "Susceptible to aging in that it dries out forming hairline cracks which can lead to micro current leakage (i.e. electrical 'ticking' faults)"

    There are ways to do corrosion-resistant contacts without precious metals; the automotive industry has solved this problem. The alloys aren't simple; here's one used for under-hood automotive connectors. Copper, iron, magnesium, and phosphorus, with upper limits on tin, zinc, nickel, lead, and manganese. But avionics connectors are usually gold plated; it doesn't add that much cost. And Russia is a major exporter of gold.

    The article doesn't go far enough. OK, the connectors corroded. Why? Wrong alloy? Plating failure? Wear from too many connector insertions? Was the spec wrong, or were the cables not made to spec?

  19. Indeed, how many russion casualties have there bee by SmallFurryCreature · · Score: 5, Insightful

    Tell me, how many casualties have the russians had in the last decade, even last two decades? This was in the days of Mir, when the russians maintained a continues space pressence year after year and the US was out of space for year after year for blowing up space shuttles.

    So whose tech is behind whose? The ISS didn't plunge out of the sky when the Space Shuttle was not available, apparently the russian capability is more then enough to operate it.

    And finally, who build the de-humidefier that was the fault in the first place?

    --

    MMO Quests are like orgasms:

    You may solo them, I prefer them in a group.