Slashdot Mirror


Upside-Down Sensors Caused Proton-M Rocket Crash

Michi writes "According to Anatoly Zak, the crash of the Russion Proton rocket on 1 July was apparently caused by several angular velocity sensors having been installed upside down. From the source: 'Each of those sensors had an arrow that was supposed to point toward the top of the vehicle, however multiple sensors on the failed rocket were pointing downward instead.' It seems amazing that something as fundamental as this was not caught during quality control. Even more amazing is that the design of the sensors permits them to be installed in the wrong orientation in the first place. Even the simplest of mechanical interlocks (such as a notch at one end that must be matched with a corresponding projection) could have prevented the accident." A review of the quality control procedures used by the contractors responsible is underway.

34 of 323 comments (clear)

  1. vodka and work don't mix by alen · · Score: 4, Interesting

    being from there i bet half the people working on this came to work drunk and/or hung over most days

  2. QA is not the problem by Anonymous Coward · · Score: 3, Insightful

    Murphy's Law is still in effect. Like the snippet says make sure that they can only be installed one way mechanically, because you won't catch 100% of the errors in QA.

    1. Re:QA is not the problem by jeffmeden · · Score: 5, Insightful

      What seems more amazing is that a simple software check pre-launch (i.e. "do all the sensors think they are pointed up?") was not part of the SOP. Given that their exact function is orientation detection, skipping the opportunity for self-test via that function is somewhat baffling.

      Obligatory: It's not rocket science!

    2. Re:QA is not the problem by oobayly · · Score: 3, Interesting

      Never underestimate the ingenuity that people are capable of in order to install something wrong. Somebody in our office forced (yes, forced) a Xerox Phaser ink block in the the slot the wrong way round. The thing is basically a shape sorter that a toddler is capable of understanding.

    3. Re:QA is not the problem by jrumney · · Score: 4, Insightful

      The greatest pleasure my toddler ever got from his shape sorter was when he discovered that the 3 could be forced through the hole for the C. Never underestimate the satisfaction a disgruntled office worker gets from jamming the ink block into the printer the wrong way around.

    4. Re:QA is not the problem by TheCarp · · Score: 5, Informative

      Amusingly, when someone actually attempted to track down who murphy was, and where the law came from.... http://en.wikipedia.org/wiki/Murphys_law

      Edward Murphy proposed using electronic strain gauges attached to the restraining clamps of Stapp's harness to measure the force exerted on them by his rapid deceleration. Murphy was engaged in supporting similar research using high speed centrifuges to generate g-forces. Murphy's assistant wired the harness, and a trial was run using a chimpanzee.

      The sensors provided a zero reading; however, it became apparent that they had been installed incorrectly, with each sensor wired backwards. It was at this point that a disgusted Murphy made his pronouncement

      So this is potentially, very much related to the original usage.

      --
      "I opened my eyes, and everything went dark again"
    5. Re:QA is not the problem by Athanasius · · Score: 4, Insightful

      My reading of 'angular velocity sensor' is that they're meant to sense rotation. If you're sat stationary on the pad there is no such rotation and thus you'll get a 'correct' zero reading. You'd have to perform such a test during some known movements of the rocket (part).

    6. Re:QA is not the problem by Baloroth · · Score: 5, Informative

      What seems more amazing is that a simple software check pre-launch (i.e. "do all the sensors think they are pointed up?") was not part of the SOP. Given that their exact function is orientation detection, skipping the opportunity for self-test via that function is somewhat baffling.

      Obligatory: It's not rocket science!

      The sensors in question were for angular velocity. Given that pre-launch the craft doesn't have any (peculiar) angular velocity, the sensors would return the correct results (zero) no matter how they were installed.

      --
      "None can love freedom heartily, but good men; the rest love not freedom, but license." --John Milton
    7. Re:QA is not the problem by KiloByte · · Score: 5, Interesting

      An old joke:

      A militia (communist police) station has been ordered to conduct an intelligence test. It consisted of a board with three holes: a circle, a triangle and a square, and three corresponding blocks. The next days, the commandant announces: I'm very proud of our station: all of you passed the test! 5% have shown exceptional intelligence, 95% exceptional strength!

      --
      The creatures outside looked from Alt-Right to Antifa; but already it was impossible to say which was which.
    8. Re:QA is not the problem by LordLimecat · · Score: 4, Funny

      Ive seen RAM modules installed backwards. "Wait!", you say, "Isnt there a notch which prevents that?" Well, yes, there WAS a notch...

    9. Re:QA is not the problem by Brett+Buck · · Score: 3, Interesting

      IT IS NOT ZERO, its 0.004187 degrees per second around some vector. The Earth is turning!

    10. Re:QA is not the problem by smellsofbikes · · Score: 3, Informative

      Amusingly, when someone actually attempted to track down who murphy was, and where the law came from.... http://en.wikipedia.org/wiki/Murphys_law

      Edward Murphy proposed using electronic strain gauges attached to the restraining clamps of Stapp's harness to measure the force exerted on them by his rapid deceleration. Murphy was engaged in supporting similar research using high speed centrifuges to generate g-forces. Murphy's assistant wired the harness, and a trial was run using a chimpanzee.

      The sensors provided a zero reading; however, it became apparent that they had been installed incorrectly, with each sensor wired backwards. It was at this point that a disgusted Murphy made his pronouncement

      So this is potentially, very much related to the original usage.

      If I remember right, the way a wheatstone strain gauge is set up, there are four ways to connect it. One is right, two are wrong but give you half the resolution you expected -- so you get data, just lousy data -- and one is completely wrong and you get no data whatsoever. It was hooked up in the completely wrong configuration. That was what made him so mad: there was only a 25% chance it would get hooked up in the completely worthless configuration, but that's what happened.

      --
      Nostalgia's not what it used to be.
  3. Wrong hemisphere by jovius · · Score: 4, Funny

    Should have launched from Australia.

    1. Re:Wrong hemisphere by ArcadeMan · · Score: 3, Funny

      That's the problem. Some parts were made by subcontractors in Australia, where up is down, rats are as tall as humans and hop around and flat-chested 30-years-old women are classified as underage teenagers.

  4. Re:The quality conrol problems... by Anonymous Coward · · Score: 3, Funny

    Quality Control in Russia basically consists of hitting it with a mallet, and if it doesn't fall apart on impact, it passes.

  5. Shades of the US "Genesis" sample return probe... by Ellis+D.+Tripp · · Score: 4, Interesting

    which plowed into the desert floor without deploying any parachutes because a G-switch was installed backwards...

    http://www.universetoday.com/73/genesis-accident-report-released/

    --
    Remember "News for Nerds, Stuff that Matters"? Help make it a reality again! http://soylentnews.org
  6. Re:In Soviet Russia by RenderSeven · · Score: 5, Funny

    Perhaps the thinking is, as long as the arrow isnt pointed at you it's probably safe.

  7. Re:(The Real) Murphy's Law strikes again! by MickLinux · · Score: 5, Interesting

    Yes, the real, original Murphy's law apparently came from Col. Stapp, who was testing rocket sleds for the rocket program.

    I should note that the putative original Murphy's Law reads, "If there are two or more ways to do something, and one of those ways can result in a catastrophe, then someone will do it." . The website goes on to say "This is a principle of defensive design, cited here because it is usually given in mutant forms less descriptive of the challenges of design for lusers. For example, you don't make a two-pin plug symmetrical and then label it `THIS WAY UP'; if it matters which way it is plugged in, then you make the design asymmetrical."

    Highly appropriate to the topic, I might say. If only they had labeled, with the arrow, the words "up", and put another arrow down, with the letters "dn" for "down", then none of this would have happened.

    For those who wish to nit-pick my attention to detail and editing, also, I will for further irony include the wikipedia link, as well: http://wikipedia.org/wiki/Muphry%27s_law

    --
    Correct Horse Battery Staple: 72 bits of entropy. Enter "Correct H" into google. When it generates the phrase, that's
  8. Re:The quality conrol problems... by PetiePooo · · Score: 5, Informative

    ...aren't so amazing when you look at the track record of Russian manufacturing.

    Before we Americans point too many fingers, let's not forget NASA is not immune to similar mistakes.

  9. Re:(The Real) Murphy's Law strikes again! by PetiePooo · · Score: 5, Informative

    My favorite is Cole's Law...

  10. Duck Dodgers in the 24 1/2th Century by dkleinsc · · Score: 3, Funny

    "Whoopth, I had the thilly thing in reverthe!"

    --
    I am officially gone from /. Long live http://www.soylentnews.com/
  11. I have seen something similar. by 140Mandak262Jamuna · · Score: 3, Interesting
    Back in the day when I was with the ministry of defense we lost a vehicle due to an error like this. They had changed the vendor for the gyros of the roll sensor. The new gyros had the voltages in the reverse sense. It is possible one vendor was European and the other was American. They wired it according to the sense of the old vendor. So the control input to the ailerons would add to the roll instead of counteracting it. The RPV crashed 1.5 seconds after launch.

    In the postmortem the flight director started with, "... we sadly lost the vehicle after a flight of 1.5 seconds ...". The mission director interrupted, "What flight? The damned thing had a 6000 Kg[sic][*] rocket booster. You can put it under a 3 ton rock and it will 'fly' for more than 2 seconds..."

    [*]He should have said 6000 Kgf-sec, because that was the impulse delivered by the twin rocket boosters each 1500 Kgf thrust burning for 2 seconds.

    --
    sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
  12. Mars orbital failure by Okian+Warrior · · Score: 5, Insightful

    The US once sent a probe all the way to mars, only to have it fail because the ground computer was in imperial units while the orbiter was in SI units.

    Getting everything correct is hard... really hard. For most projects you have elaborate "fail gracefully" modes which rely on external agents to notice the problem and take action. A doctor or pilot can take appropriate action, but it's hard to do with rockets.

    For comparison, I wrote the software for the altimeter that goes into some 747 aircraft. Total of about 21,000 lines of C, about 40% comments so figure 12,000 lines of code. The testers (and I) worked really hard to find all bugs in the system, knowing that a mistake could knock a plane out of the sky. There were elaborate internal checks both in software and process, and Boeing did their own testing on top of ours. Everything passed, all requirements were met, things looked good.

    The device had 1 bug, found after installation. A software typo which wasn't caught by QA even though it had a specific testing requirement. No one was negligent, it just slipped by despite best efforts.

    Multiply this by all the devices in an aircraft, and add in the other engineering disciplines like electronics and mechanical. It's really hard to get everything right all at once, and on the first try.

  13. Re:The quality conrol problems... by ArcadeMan · · Score: 3, Insightful

    There's a difference between true communism and corrupt, dictatorial regimes.

    For those too lazy to click on the link:

    Communism (from Latin communis - common, universal) is a revolutionary socialist movement to create a classless, moneyless and stateless social order structured upon common ownership of the means of production, as well as a social, political and economic ideology that aims at the establishment of this social order.

    A perfect example of true communism applied to a specific field would be open-source software.
    A perfect example of corruption (on the capitalism side, too) would be Microsoft, threatening computer manufacturers about increasing the cost of Windows if they offered Linux options.

  14. heh by wbr1 · · Score: 4, Funny

    In Soviet Russia, snesors installed correctly, rocket installed upside down.

    --
    Silence is a state of mime.
  15. Re:The quality conrol problems... by LordLimecat · · Score: 3, Insightful

    Communism (n) - an unattainable standard that is constantly held up as a model of perfection despite having no functional real world example past or present. Related entries: No True Scotsman; Ivory Tower Intellectualism.

  16. Re:(The Real) Murphy's Law strikes again! by akh · · Score: 5, Funny

    If only they had labeled, with the arrow, the words "up", and put another arrow down, with the letters "dn" for "down", then none of this would have happened.

    Except that "dn" upside-down is indistinguishable from "up". Murphy strikes again?

    --
    Accept Eris as your Fnord and personally sate her
  17. Re:The quality conrol problems... by 0123456 · · Score: 4, Insightful

    Communism (n) - an unattainable standard that is constantly held up as a model of perfection despite having no functional real world example past or present.

    Sorry, but that's nonsense. All you need to do to create perfect communism is kill everyone else so no-one can disagree with you (you can't just kill the ones who disagree, because the others might only be pretending to agree).

    Stalin made a pretty good attempt, but didn't quite succeed.

  18. Contractors? by whitroth · · Score: 3, Interesting

    The Russians are using contractors, now?

    On the other hand, they seem to be doing vastly better than the US these days - we have NO WAY to put someone in orbit (unless the Pentagon's got a black program).

    We also had Challenger and Columbia. And on the latter note, I'll add that I believe my late ex's analysis, rather than the "it's falling insulation" answer. She was an engineer, and worked at the Cape for 17 years, including on the Shuttle, and she thought that some of the inspections that were supposed to be done were *not* being done, or not being done as frequently as they were supposed to have been... and that the hydraulic lines broke due to stress corrosion microcracking, and there went the aerilons.

    So, how many astronauts/cosmonauts have the Russians lost lately?

                      mark

  19. Re:The quality conrol problems... by jellomizer · · Score: 3, Insightful

    It looked good on Paper doesn't it.

    However it rarely works for a long time or with a lot of people.

    The problem is that we live in a world of scarcity, we can't get all that stuff that we want, or need. The communist system tries to make everything equal, however that means everyone will be living in scarcity, and not really having what they need or want. Because everyone will be wanting, it will open the door for someone(s) to cheat the system and try to get more, once they have more they will hold on to it. And the system begins to fail.

    Software like Open Source tends to work better, because there isn't a limit in supply. You can copy share make a copy of the copy and continue on and on. There is no scarcity in the Open Source Model.

    --
    If something is so important that you feel the need to post it on the internet... It probably isn't that important.
  20. Re:The quality conrol problems... by Immerman · · Score: 4, Insightful

    >That would be like saying that disaster relief volunteers are an example of communism
    Actually that sounds about right to me. "From each according to their ability, to each according to their need" and all that.

    Most family households also run on at least partially communist principles. Luxuries may need to be at least partially earned (or not, plenty of douches with entitlement issues out there), but it's a sad family where everyone's *needs* aren't taken care of first.

    The problem with communism seems to be that it doesn't seem to scale well beyond the tribe/monastery/commune level. Once the population gets too large to allow for effective communal decision making, communal ownership tends to become de-facto ownership by the decision makers, massively exacerbating the problem of corruption.

    --
    --- Most topics have many sides worth arguing, allow me to take one opposite you.
  21. Re:The quality conrol problems... by wagnerrp · · Score: 4, Interesting

    Actually, that's wrong. Capitalism is designed with corruption and greed in mind. Greed motivates entities to perform better in the market and get more stuff. Corruption is dealt with by entities shifting to competitors who are screwing them less. Capitalism's failure is in assuming all involved entities are sufficiently intelligent to be aware of when they're being screwed, and principled enough to forgo what they want to avoid being screwed.

  22. Re:The quality conrol problems... by LordLimecat · · Score: 3, Insightful

    Democracy /capitalism "work" even if you dont have a "pure" implementation.

    Communism has never worked in any of its forms; its just gotten millions killed in purges and famines, and left nations in a crippled, dysfuncitonal state even decades later. The cry has always been that it hasnt worked because it wasnt implemented in a pure enough form, hence my "no true scotsman" comment.

    Capitalist / democratic states, however, continue to be represented by every major world power. China is becoming a major power precisely by embracing a functioning economic system that looks and smells an awful lot like capitalism.

  23. Re:The quality conrol problems... by Minwee · · Score: 3, Insightful

    All you need to do to create perfect communism is kill everyone else so no-one can disagree with you (you can't just kill the ones who disagree, because the others might only be pretending to agree).

    Stalin made a pretty good attempt, but didn't quite succeed.

    By an incredible coincidence, that's also the way to create a perfectly free market with no government intervention.