Slashdot Mirror


Why ISS Computers Failed

Geoffrey.landis writes "It was only a small news item four months ago: all three of the Russian computers that control the International Space Station failed shortly after the Space Shuttle brought up a new solar array. But why did they fail? James Oberg, writing in IEEE Spectrum, details the detective work that led to a diagnosis." The article has good insights into the role the ISS plays as a laboratory for US-Russian technology cooperation — something that is likely to be crucial in any manned Mars mission.

31 of 324 comments (clear)

  1. Urgh. by Airconditioning · · Score: 5, Insightful

    The article reeked of condesension towards the Russians. It's no way to report on your partners in space.

    1. Re:Urgh. by Anonymous Coward · · Score: 4, Insightful

      Yup. OK, it's a design flaw. We have been, and still are, capable of doing things just as bad, if not far worse. Look at the Shuttle fiascos.

      This item is hugely biased. It looks to me like a simple case of corrosion, which could easily have been patched up if it happened on a Mars flight. The engineers and crew all seemed to work well together, and the Russians were the ones who sorted the problem.

      I don't know if the Russian Program Managers got all political against us, but the item, written by a retired NASA manager, sure as hell gets political against the Russians. He's right in one thing - the managers need to stop getting political, and I suggest he starts with himself!

      It's just as well he's retired - looks like he's fighting long lost battles against cooperation with the Russians and Europeans.

    2. Re:Urgh. by Ethanol-fueled · · Score: 5, Insightful

      Hell yeah. Mod parent up. The real heroes are in space cooperating and solving problems.
      Seriously, all of that political cold war-era cockwaving should stop.

    3. Re:Urgh. by shutdown+-p+now · · Score: 2, Insightful

      Seriously, all of that political cold war-era cockwaving should stop.
      Given that we are clearly moving into another cold war period, why would it?
    4. Re:Urgh. by JoelKatz · · Score: 3, Insightful

      Absolutely.

      "It is dismaying that after decades of experience with manned space stations, Russian space engineers still couldn't keep unwanted condensation at bay."

      That's a bunch of crap. That's like saying it's dismaying that McDonald's has served billions of burgers and still can't figure out how to make them healthy.

      Condensation is "still" a problem because it's one of the big and tricky ones. To get rid of the condensation, you have to get rid of the people.

  2. Duct Tape by istartedi · · Score: 4, Insightful

    They also decided to rig a thermal barrier out of a surplus reference book and all-purpose gray tape

    Almost certainly, this was the duct tape we all know and love. They probably thought it was better not to actually say that, though. Pretty funny. And as an added side-benefit, they should be safe from terrorists.

    --
    For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
  3. Redundancy != Safety by quanticle · · Score: 5, Insightful

    I think NASA should have learned this lesson by now. After all, the Challenger disaster showed this principle as well. In that case, the same cold temperature that weakened the primary seal on the solid rocket booster weakened the secondary as well, sapping its ability to provide redundant backup. In this case, the same condensation affected all three computers equally.

    Its troubling to see them taking shortcuts on safety and redundancy, when such measures have resulted in loss of life before. How hard would it have been to have had three shut-off cables?

    --
    We all know what to do, but we don't know how to get re-elected once we have done it
    1. Re:Redundancy != Safety by khallow · · Score: 4, Insightful

      Its troubling to see them taking shortcuts on safety and redundancy, when such measures have resulted in loss of life before. How hard would it have been to have had three shut-off cables?

      At first, I was nodding in agreement. But then I realized, how do you find out when you've built in hidden single points of failure? Everyone knows that a single point of failure is bad. Hence, the ones that get into a space station weren't intended (or were due to shoddy work). One way to find them is to use the equipment in a real situation and vet it when it breaks. Exactly what they did. Now that they know this is a problem, they can fix it.
  4. Re:The REAL reason they failed by ozmanjusri · · Score: 1, Insightful
    Vista just doesn't get any breaks on this site... ever

    You're joking, I hope?

    Every discussion of Vista is FULL of astroturfers defending the OS, and they're always modded up. It's almost impossible to discuss its real flaws because of all the Microsoft-sponsored noise.

    --
    "I've got more toys than Teruhisa Kitahara."
  5. Proper debugging technique by dd1968 · · Score: 5, Insightful
    These computers functioned for months or years. When they failed, the right question to ask first was "what has changed?" This is exactly what the Russians did. According to the author the Russians first considered potential causes stemming from the newly installed solar power wing, the visiting shuttle, and the expanded station structure (the reason for the shuttle being there). One conclusion is that they were pointing the finger at NASA and playing the blame game. Another is that they were doing what good engineers anywhere would do to debug the problem.

    The author is obviously way more qualified than I to assess the situation and he may well be right but from the content of the article I came away thinking, wow, I would have looked first at all the recent changes to the station and the power supply too.

    1. Re:Proper debugging technique by DNS-and-BIND · · Score: 3, Insightful
      I see you have never dealt with Russians. The ones in their space program are especially tetchy about taking ANY blame whatsoever. Their equipment is always perfect, and the foreign equipment MUST be the problem. You know, how when there's a problem, you kind of step back for a second and analyze the entire situation? That's what NASA does. The Russians merely blame the first thing they can think of. Then, when that's disproven, they have a lot of other proposed explanations, none of which involve the failure of Russian equipment. It's even worse when there is a semi-plausible event like the new solar panel.

      Look, the Russians as people are all right. But their management in the space program is obsessed with face. They feel that admitting any faults demeans the Russian nation and the Russian people. You can laugh but that's how it is.

      --
      Shutting down free speech with violence isn't fighting fascism. It IS fascism!
    2. Re:Proper debugging technique by giafly · · Score: 4, Insightful

      I see you have never dealt with Russians. The ones in their space program are especially tetchy about taking ANY blame whatsoever. Their equipment is always perfect, and the foreign equipment MUST be the problem.
      I see you have never worked in the computer industry, if you think this mindset is unique to Russians. Actually it is universal.
      --
      Reduce, reuse, cycle
    3. Re:Proper debugging technique by Anonymous Coward · · Score: 1, Insightful

      I see you have never dealt with Americans. The ones in their space program are especially tetchy about taking ANY blame whatsoever. Their equipment is always perfect, and the foreign equipment MUST be the problem. You know, how when there's a problem, you kind of step back for a second and analyze the entire situation? That's what Russians do. The NASA merely blames the first thing it can think of. Then, when that's disproven, they have a lot of other proposed explanations, none of which involve the failure of American equipment. It's even worse when there is a semi-plausible event like Russian equipment on board.

      Look, the Americans, as people, are all right. But their management in the space program is obsessed with face. They feel that admitting any faults demeans the American pride and the American people. You can laugh but that's how it is.

  6. It's interesting... by JustShootMe · · Score: 4, Insightful

    That for all of the controls and quality control required of mission critical hardware such as this, it still comes down to:

    1) unexpected failure modes
    2) political battles

    Which really isn't a whole lot different than 1) the unexpected failure modes I see every day at work, and 2) the political wrangling (fingerpointing) that takes place when they happen. Apparently NASA and its Russian equivalent are no better than any old software company.

    The lesson being, people are people, and people are still the ones that design these things.

    --
    For linux tips: http://www.linuxtipsblog.com
  7. Hate to break it to you... by patio11 · · Score: 3, Insightful

    ... but for equipment which is all critical, all essentially one-of-a-kind, and all lethal if compromised, there are only two safety states: failed and "has not failed... yet".

  8. Jingoism by QuickFox · · Score: 1, Insightful
    FTA:

    It is dismaying that after decades of experience with manned space stations, Russian space engineers still couldn't keep unwanted condensation at bay. But what's worse is that they designed circuitry that would allow one spot of corrosion to fell a supposedly triply redundant control computer complex. I find it more dismaying that an otherwise seemingly adult and mature article writer feels such an urge to childishly emphasize blame. What is it with this childish American and Russian jingoism? If blame is so important, can't you people at least blame the engineers and not the nationality?
    --
    Terrorists can't threaten a country's freedom and democracy. Only lawmakers and voters can do that.
  9. I hope they don't by khallow · · Score: 4, Insightful

    The article has good insights into the role the ISS plays as a laboratory for US-Russian technology cooperation -- something that is likely to be crucial in any manned Mars mission.

    No offense to Russia or the US, both who produce good space gear, but technology cooperation is probably a bad idea unless it is tested more thoroughly than in the ISS. The ISS is a great example of how to screw up international cooperation. The station has been delayed for more than a decade (and cost NASA around $50 billion so far) due to redesign and indecision, reliance on a single launch vehicle for key components (the Shuttle), and the inclusion of the Russians. There are parts of the station that can only communicate with the Russians and parts that can only communicate with NASA. Aside from basic utility hookup (electricity), there's no connection between the different parties on the ISS (at least between the Russians and NASA, the ESA and Japanese parts might work better with NASA's stuff). And if you want to make changes that affect more than one party, it becomes by default an international issue. Finally, there's no easy way to transfer ownership. NASA's communication system is integral (TDRSS) to the NASA parts and is also a national secret (so I understand). So the communication system can't be transfered to another party like the Russians or the ESA.

    If there's any international cooperation between space agencies, it probably should be at a rather trivial and manageable level. Say including foreign astronauts or using off the shelf equipment that is know to work under the circumstances.

    1. Re:I hope they don't by khallow · · Score: 2, Insightful

      Actually, they're feeding the NASA supply chain. For the money NASA spent on the ISS, they could have built 3-5 ISS's, maybe more if they eliminated the dependency on the Shuttle and used Titan IV's instead. This little bit of testing came at a very high price.

  10. Re:The REAL reason they failed by QuickFox · · Score: 3, Insightful

    Microsoft astroturfers can get mod points just like everybody else. Don't give much importance to mod points, that way you'll feel much better.

    --
    Terrorists can't threaten a country's freedom and democracy. Only lawmakers and voters can do that.
  11. Re:The REAL reason they failed by FoolsGold · · Score: 4, Insightful

    Are you honestly saying that anyone who thinks Vista is decent is a MS shrill?

    Why? Is defending a MS operating system for honest reasons impossible to believe anymore?

  12. Re:The REAL reason they failed by CodeBuster · · Score: 2, Insightful

    You mean "downgraded" right?

  13. Wiring corrosion? by Animats · · Score: 4, Insightful

    I'm surprised that connector corrosion would be a problem. Aviation has a long history of wire problems, but gold-plating connectors seems to be a stable solution to that problem. The ISS uses Kapton wire, which was popular in the 1980s and is lightweight and tough. But that material is hygroscopic and now banned by the USAF, US Navy, Boeing, etc. "Susceptible to aging in that it dries out forming hairline cracks which can lead to micro current leakage (i.e. electrical 'ticking' faults)"

    There are ways to do corrosion-resistant contacts without precious metals; the automotive industry has solved this problem. The alloys aren't simple; here's one used for under-hood automotive connectors. Copper, iron, magnesium, and phosphorus, with upper limits on tin, zinc, nickel, lead, and manganese. But avionics connectors are usually gold plated; it doesn't add that much cost. And Russia is a major exporter of gold.

    The article doesn't go far enough. OK, the connectors corroded. Why? Wrong alloy? Plating failure? Wear from too many connector insertions? Was the spec wrong, or were the cables not made to spec?

  14. Re:It's not surprising by wishmechaos · · Score: 2, Insightful

    Every time people mod up a clippy post, a little part of me dies.

  15. Indeed, how many russion casualties have there bee by SmallFurryCreature · · Score: 5, Insightful

    Tell me, how many casualties have the russians had in the last decade, even last two decades? This was in the days of Mir, when the russians maintained a continues space pressence year after year and the US was out of space for year after year for blowing up space shuttles.

    So whose tech is behind whose? The ISS didn't plunge out of the sky when the Space Shuttle was not available, apparently the russian capability is more then enough to operate it.

    And finally, who build the de-humidefier that was the fault in the first place?

    --

    MMO Quests are like orgasms:

    You may solo them, I prefer them in a group.

  16. Life will find a way by TyroPyro · · Score: 2, Insightful

    I found it interesting that mold (fungi) was found living in the condensation. It means that despite the what I presume is a strict level of sterilization and sanitation for both Astronauts and equipment headed to the ISS, some spores still made it up and began to replicate in this one little area of opportunity.

  17. Re:Hmmmm. by CharlieG · · Score: 3, Insightful

    I think NASA's BIG mistake (pun intended) was designing the modules such that they could ONLY be lifted by the shuttle, instead of the then Titan's, or today's Delta/Atlas heavy lift versions, particularly post Challenger, when all the commercial stuff got moved off the shuttle.

    If they had designed the modules for multiple lift modes, if one was NOT operational, the odds are the other would be. THAT is true redundency - 2 totally different systems, each capable of doing the job

    --
    -- 73 de KG2V For the Children - RKBA! "You are what you do when it counts" - the Masso
  18. Re:Indeed, how many russion casualties have there by Anonymous Coward · · Score: 1, Insightful

    I'm not really pro-American at all. I think the Russian program is actually superior, the shuttle's just too bloated and complex.

    The one thing you've got to give the Americans is that they're prepared to admit when they've got casualties. I find it hard to believe that Russians didn't attempt to launch people previously and just didn't report the failures.

  19. Re:The REAL reason they failed by Woy · · Score: 4, Insightful

    > Is defending a MS operating system for honest reasons impossible to believe anymore?

    We don't do honest here. We do technically sound.

    --
    "If God created us in his own image we have more than reciprocated." - Voltaire
  20. Re:Duct tape saves the day! by hey! · · Score: 2, Insightful

    Over the years, I have come to realize that my entire career could have been inspired by duct tape.

    Two studies related to duct tape have been reported recently. The first was a government study of various commercial products for affixing insulation to HVAC systems, which found that every product performed well over time except duct tape. The second was a study which showed that the folk remedy for warts in which you cover them with duct tape was surprisingly effective.

    There you have it: amazingly versatile, as long as you don't ask it to do what it's supposed to.

    --
    Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
  21. REAL by WED+Fan · · Score: 2, Insightful

    I love this, rather than discuss the real issues, /. can't even talk about other computers without bashing MS.

    --
    Politics is the art of looking for trouble, finding it everywhere, diagnosing it incorrectly and applying the wrong fix.
  22. Re:The REAL reason they failed by encoderer · · Score: 4, Insightful

    You missed one important part...

    Millions, nay Tens of Millions of people give Microsoft and their products "the time of day." People who have no dogmas or political agendas when it comes to computing. People who just see a computer and its software as a tool to get their desired job done. And not just MBA or Administration types, but also millions of software developers and network administrators and such.

    I don't think Windows is perfect, but I also don't think OSX is perfect nor do I think that Linux or any flavor of Unix is perfect. I do think that the O^n usefulness of the Windows install base provides so much opportunity that it ends up offering the most value to businesses and consumers.

    And with regard to their "self serving" ways... many on slashdot are anti-business or at least anti-corporation. They adopt the FSF malarkey that all code should be given away free. I put food on my family's table by developing software and the notion that it should be given away free just misses the mark. Market-based economics can bring out the best in innovation, which is why America has some of the highest paid and most productive workers in the world.

    Slashdot is full of idealistic college students and 20-somethings (of which I am a part) who think that corporations are "evil" and that we should all wear birkenstocks and eat crunchy granola and spend our days writing software that solves a problem that's already been solved on a Windows platform and then give it away for free just so we can say we fought the good fight. It's naive. Say what you want about Microsoft, but that company, and the efforts of billg have made THOUSANDS of people millionaires and probably a handful of billionaires, too. Many of those people took that money and started their own software companies solving their own unique, novel problems, and on their own hiring employees and fueling the economy and probably making a lot of those people millionaires, too, who perpetuate it.

    Business is good for all of us. Economic success and security is good for America.