Slashdot Mirror


Upside-Down Sensors Caused Proton-M Rocket Crash

Michi writes "According to Anatoly Zak, the crash of the Russion Proton rocket on 1 July was apparently caused by several angular velocity sensors having been installed upside down. From the source: 'Each of those sensors had an arrow that was supposed to point toward the top of the vehicle, however multiple sensors on the failed rocket were pointing downward instead.' It seems amazing that something as fundamental as this was not caught during quality control. Even more amazing is that the design of the sensors permits them to be installed in the wrong orientation in the first place. Even the simplest of mechanical interlocks (such as a notch at one end that must be matched with a corresponding projection) could have prevented the accident." A review of the quality control procedures used by the contractors responsible is underway.

59 of 323 comments (clear)

  1. The quality conrol problems... by He+Who+Has+No+Name · · Score: 2, Insightful

    ...aren't so amazing when you look at the track record of Russian manufacturing.

    1. Re:The quality conrol problems... by Anonymous Coward · · Score: 3, Funny

      Quality Control in Russia basically consists of hitting it with a mallet, and if it doesn't fall apart on impact, it passes.

    2. Re:The quality conrol problems... by PetiePooo · · Score: 5, Informative

      ...aren't so amazing when you look at the track record of Russian manufacturing.

      Before we Americans point too many fingers, let's not forget NASA is not immune to similar mistakes.

    3. Re:The quality conrol problems... by ArcadeMan · · Score: 3, Insightful

      There's a difference between true communism and corrupt, dictatorial regimes.

      For those too lazy to click on the link:

      Communism (from Latin communis - common, universal) is a revolutionary socialist movement to create a classless, moneyless and stateless social order structured upon common ownership of the means of production, as well as a social, political and economic ideology that aims at the establishment of this social order.

      A perfect example of true communism applied to a specific field would be open-source software.
      A perfect example of corruption (on the capitalism side, too) would be Microsoft, threatening computer manufacturers about increasing the cost of Windows if they offered Linux options.

    4. Re:The quality conrol problems... by LordLimecat · · Score: 3, Insightful

      Communism (n) - an unattainable standard that is constantly held up as a model of perfection despite having no functional real world example past or present. Related entries: No True Scotsman; Ivory Tower Intellectualism.

    5. Re:The quality conrol problems... by Canazza · · Score: 2

      That's BS.

      There's a Titanium smelting plant in Redditch. Granted, it's Russian owned, but still, it's in the UK.

      --
      It pays to be obvious, especially if you have a reputation for being subtle.
    6. Re:The quality conrol problems... by K.+S.+Kyosuke · · Score: 2

      A perfect example of true communism applied to a specific field would be open-source software.

      You sure about that? You can't apply communism "to a specific field". That would be like saying that disaster relief volunteers are an example of communism. Also, I'm pretty sure that my means of software production, namely my brain, is not subject to common ownership. Unless somebody's borrowing it in secret whenever I'm asleep, that is.

      --
      Ezekiel 23:20
    7. Re:The quality conrol problems... by 0123456 · · Score: 4, Insightful

      Communism (n) - an unattainable standard that is constantly held up as a model of perfection despite having no functional real world example past or present.

      Sorry, but that's nonsense. All you need to do to create perfect communism is kill everyone else so no-one can disagree with you (you can't just kill the ones who disagree, because the others might only be pretending to agree).

      Stalin made a pretty good attempt, but didn't quite succeed.

    8. Re:The quality conrol problems... by jellomizer · · Score: 3, Insightful

      It looked good on Paper doesn't it.

      However it rarely works for a long time or with a lot of people.

      The problem is that we live in a world of scarcity, we can't get all that stuff that we want, or need. The communist system tries to make everything equal, however that means everyone will be living in scarcity, and not really having what they need or want. Because everyone will be wanting, it will open the door for someone(s) to cheat the system and try to get more, once they have more they will hold on to it. And the system begins to fail.

      Software like Open Source tends to work better, because there isn't a limit in supply. You can copy share make a copy of the copy and continue on and on. There is no scarcity in the Open Source Model.

      --
      If something is so important that you feel the need to post it on the internet... It probably isn't that important.
    9. Re:The quality conrol problems... by Immerman · · Score: 4, Insightful

      >That would be like saying that disaster relief volunteers are an example of communism
      Actually that sounds about right to me. "From each according to their ability, to each according to their need" and all that.

      Most family households also run on at least partially communist principles. Luxuries may need to be at least partially earned (or not, plenty of douches with entitlement issues out there), but it's a sad family where everyone's *needs* aren't taken care of first.

      The problem with communism seems to be that it doesn't seem to scale well beyond the tribe/monastery/commune level. Once the population gets too large to allow for effective communal decision making, communal ownership tends to become de-facto ownership by the decision makers, massively exacerbating the problem of corruption.

      --
      --- Most topics have many sides worth arguing, allow me to take one opposite you.
    10. Re:The quality conrol problems... by wagnerrp · · Score: 4, Interesting

      Actually, that's wrong. Capitalism is designed with corruption and greed in mind. Greed motivates entities to perform better in the market and get more stuff. Corruption is dealt with by entities shifting to competitors who are screwing them less. Capitalism's failure is in assuming all involved entities are sufficiently intelligent to be aware of when they're being screwed, and principled enough to forgo what they want to avoid being screwed.

    11. Re:The quality conrol problems... by lgw · · Score: 2

      It's just Murphy's Law in action. The original inspiration for Murphy's Law was when Murphy flew a test airplane that had instrument gauges that, just like the sensors in TFA, could physically be installed upside down, but it was completely obvious which way was right. All of the gauges in his test plane were upside down, leading him to coin the phrase that has far outlasted our memory of his career as a test pilot.

      Murphy's Law was specifically about installing aerospace instrumentation upside down.

      --
      Socialism: a lie told by totalitarians and believed by fools.
    12. Re:The quality conrol problems... by ArcadeMan · · Score: 2

      The problem is that we live in a world of scarcity, we can't get all that stuff that we want, or need. The communist system tries to make everything equal, however that means everyone will be living in scarcity, and not really having what they need or want. Because everyone will be wanting, it will open the door for someone(s) to cheat the system and try to get more, once they have more they will hold on to it. And the system begins to fail.

      This is your typical right-wing FUD right there. You don't understand the principle and then you go and make your own mind about so-called "facts".

      Communism isn't about giving everyone a fucking Hummer and a fucking five-story house with a lot the size of ten football fields and gold jewelry.

      You need to understand that making everyone equals doesn't mean there will be scarcity. Communism isn't about what people *want*, it's about what people *need*. Everybody should have free basic food, free basic housing, free potable water, free transportation, free health care, etc. You want extra? That's where the capitalism should kick in.

      With more and more jobs going to automated factories, I'm pretty sure that's where we're headed. Too lazy and want to be passive? You don't get anything extra but you can live your life peacefully without sickness and hunger. You want more than the basics? You work to get it.

    13. Re:The quality conrol problems... by LordLimecat · · Score: 3, Insightful

      Democracy /capitalism "work" even if you dont have a "pure" implementation.

      Communism has never worked in any of its forms; its just gotten millions killed in purges and famines, and left nations in a crippled, dysfuncitonal state even decades later. The cry has always been that it hasnt worked because it wasnt implemented in a pure enough form, hence my "no true scotsman" comment.

      Capitalist / democratic states, however, continue to be represented by every major world power. China is becoming a major power precisely by embracing a functioning economic system that looks and smells an awful lot like capitalism.

    14. Re:The quality conrol problems... by Minwee · · Score: 3, Insightful

      All you need to do to create perfect communism is kill everyone else so no-one can disagree with you (you can't just kill the ones who disagree, because the others might only be pretending to agree).

      Stalin made a pretty good attempt, but didn't quite succeed.

      By an incredible coincidence, that's also the way to create a perfectly free market with no government intervention.

    15. Re:The quality conrol problems... by Anonymous Coward · · Score: 2, Interesting

      I've long thought that the problem with Communism is that it assumes that people are essentially good, efficient and responsible. If the people fail to meet that standard then Communism suffers.

      Capitalism assumes that people are essentially bad, lazy and corrupt. If the people do better than that then Capitalism benefits.

  2. vodka and work don't mix by alen · · Score: 4, Interesting

    being from there i bet half the people working on this came to work drunk and/or hung over most days

    1. Re:vodka and work don't mix by tnk1 · · Score: 2

      Russians are quite capable of making some really sturdy stuff. If anything, they engineer it to stand up to operators who are often uneducated and half-drunk. That is where you get masterpieces like the AK-47 which isn't the most accurate assault rifle out there, but you can bury one in shit for a year, dig it up, clean out the mechanism a bit and it will still fire. Try that with an M-16 and you'll have a military-themed door stop.

      So, I would not be surprised if there was someone working on that Proton-M who was somehow inebriated.

      On the other hand, there's probably drunk people working on the Proton-M at all times anyway, so it probably has nothing to do with the problem.

  3. QA is not the problem by Anonymous Coward · · Score: 3, Insightful

    Murphy's Law is still in effect. Like the snippet says make sure that they can only be installed one way mechanically, because you won't catch 100% of the errors in QA.

    1. Re:QA is not the problem by jeffmeden · · Score: 5, Insightful

      What seems more amazing is that a simple software check pre-launch (i.e. "do all the sensors think they are pointed up?") was not part of the SOP. Given that their exact function is orientation detection, skipping the opportunity for self-test via that function is somewhat baffling.

      Obligatory: It's not rocket science!

    2. Re:QA is not the problem by oobayly · · Score: 3, Interesting

      Never underestimate the ingenuity that people are capable of in order to install something wrong. Somebody in our office forced (yes, forced) a Xerox Phaser ink block in the the slot the wrong way round. The thing is basically a shape sorter that a toddler is capable of understanding.

    3. Re:QA is not the problem by Anonymous Coward · · Score: 2, Funny

      there is a job waiting for them in the space optics division of Perkin-Elmer

    4. Re:QA is not the problem by jrumney · · Score: 4, Insightful

      The greatest pleasure my toddler ever got from his shape sorter was when he discovered that the 3 could be forced through the hole for the C. Never underestimate the satisfaction a disgruntled office worker gets from jamming the ink block into the printer the wrong way around.

    5. Re:QA is not the problem by TheCarp · · Score: 5, Informative

      Amusingly, when someone actually attempted to track down who murphy was, and where the law came from.... http://en.wikipedia.org/wiki/Murphys_law

      Edward Murphy proposed using electronic strain gauges attached to the restraining clamps of Stapp's harness to measure the force exerted on them by his rapid deceleration. Murphy was engaged in supporting similar research using high speed centrifuges to generate g-forces. Murphy's assistant wired the harness, and a trial was run using a chimpanzee.

      The sensors provided a zero reading; however, it became apparent that they had been installed incorrectly, with each sensor wired backwards. It was at this point that a disgusted Murphy made his pronouncement

      So this is potentially, very much related to the original usage.

      --
      "I opened my eyes, and everything went dark again"
    6. Re:QA is not the problem by Athanasius · · Score: 4, Insightful

      My reading of 'angular velocity sensor' is that they're meant to sense rotation. If you're sat stationary on the pad there is no such rotation and thus you'll get a 'correct' zero reading. You'd have to perform such a test during some known movements of the rocket (part).

    7. Re:QA is not the problem by Baloroth · · Score: 5, Informative

      What seems more amazing is that a simple software check pre-launch (i.e. "do all the sensors think they are pointed up?") was not part of the SOP. Given that their exact function is orientation detection, skipping the opportunity for self-test via that function is somewhat baffling.

      Obligatory: It's not rocket science!

      The sensors in question were for angular velocity. Given that pre-launch the craft doesn't have any (peculiar) angular velocity, the sensors would return the correct results (zero) no matter how they were installed.

      --
      "None can love freedom heartily, but good men; the rest love not freedom, but license." --John Milton
    8. Re:QA is not the problem by KiloByte · · Score: 5, Interesting

      An old joke:

      A militia (communist police) station has been ordered to conduct an intelligence test. It consisted of a board with three holes: a circle, a triangle and a square, and three corresponding blocks. The next days, the commandant announces: I'm very proud of our station: all of you passed the test! 5% have shown exceptional intelligence, 95% exceptional strength!

      --
      The creatures outside looked from Alt-Right to Antifa; but already it was impossible to say which was which.
    9. Re:QA is not the problem by LordLimecat · · Score: 4, Funny

      Ive seen RAM modules installed backwards. "Wait!", you say, "Isnt there a notch which prevents that?" Well, yes, there WAS a notch...

    10. Re:QA is not the problem by Baloroth · · Score: 2

      Sure, if this was a vertically mounted accelerometer. It was not: it was an angular velocity sensor (I believe this is the technical specification page, although I could be wrong). The angular velocity of the craft on the launchpad is zero. If it's not, you have much much bigger problems than an improperly mounted sensor.

      --
      "None can love freedom heartily, but good men; the rest love not freedom, but license." --John Milton
    11. Re:QA is not the problem by dpidcoe · · Score: 2

      When I worked in IT, the office movers were notorious for attempting to hook up computers (we told them not to) the wrong way. They destroyed several video cards by connecting the trapezoid shaped dell matrix connector (it splits off into 2 DVI connectors) upside down. Somehow they forced it on enough that the screws could bite, then cranked it down as hard as they could, completely demolishing the interface between the connector and the card. I don't think I could have done a better job had I taken a sledgehammer to it.

    12. Re:QA is not the problem by Brett+Buck · · Score: 3, Interesting

      IT IS NOT ZERO, its 0.004187 degrees per second around some vector. The Earth is turning!

    13. Re:QA is not the problem by smellsofbikes · · Score: 3, Informative

      Amusingly, when someone actually attempted to track down who murphy was, and where the law came from.... http://en.wikipedia.org/wiki/Murphys_law

      Edward Murphy proposed using electronic strain gauges attached to the restraining clamps of Stapp's harness to measure the force exerted on them by his rapid deceleration. Murphy was engaged in supporting similar research using high speed centrifuges to generate g-forces. Murphy's assistant wired the harness, and a trial was run using a chimpanzee.

      The sensors provided a zero reading; however, it became apparent that they had been installed incorrectly, with each sensor wired backwards. It was at this point that a disgusted Murphy made his pronouncement

      So this is potentially, very much related to the original usage.

      If I remember right, the way a wheatstone strain gauge is set up, there are four ways to connect it. One is right, two are wrong but give you half the resolution you expected -- so you get data, just lousy data -- and one is completely wrong and you get no data whatsoever. It was hooked up in the completely wrong configuration. That was what made him so mad: there was only a 25% chance it would get hooked up in the completely worthless configuration, but that's what happened.

      --
      Nostalgia's not what it used to be.
    14. Re:QA is not the problem by pspahn · · Score: 2

      When I was about 12, I took a knife and shaved off the corners of a molex connector and attached it an old hard disk that had tons of bad sectors. I just wanted to see what would happen.

      While interesting to a 12 year old, it was nothing more than a couple pops, some smoke, and a little bit of melted plastic. It was at this very moment that I learned everything I ever needed to know about computers.

      --
      Someone flopped a steamer in the gene pool.
    15. Re:QA is not the problem by PPH · · Score: 2

      can only be installed one way mechanically,

      Won't help. The assembly notes say, "Beat to fit. Paint to match."

      --
      Have gnu, will travel.
  4. (The Real) Murphy's Law strikes again! by Oloryn · · Score: 2

    Wasn't something like this responsible for the formulation of Murphy's law?

    1. Re:(The Real) Murphy's Law strikes again! by MickLinux · · Score: 5, Interesting

      Yes, the real, original Murphy's law apparently came from Col. Stapp, who was testing rocket sleds for the rocket program.

      I should note that the putative original Murphy's Law reads, "If there are two or more ways to do something, and one of those ways can result in a catastrophe, then someone will do it." . The website goes on to say "This is a principle of defensive design, cited here because it is usually given in mutant forms less descriptive of the challenges of design for lusers. For example, you don't make a two-pin plug symmetrical and then label it `THIS WAY UP'; if it matters which way it is plugged in, then you make the design asymmetrical."

      Highly appropriate to the topic, I might say. If only they had labeled, with the arrow, the words "up", and put another arrow down, with the letters "dn" for "down", then none of this would have happened.

      For those who wish to nit-pick my attention to detail and editing, also, I will for further irony include the wikipedia link, as well: http://wikipedia.org/wiki/Muphry%27s_law

      --
      Correct Horse Battery Staple: 72 bits of entropy. Enter "Correct H" into google. When it generates the phrase, that's
    2. Re:(The Real) Murphy's Law strikes again! by PetiePooo · · Score: 5, Informative

      My favorite is Cole's Law...

    3. Re:(The Real) Murphy's Law strikes again! by akh · · Score: 5, Funny

      If only they had labeled, with the arrow, the words "up", and put another arrow down, with the letters "dn" for "down", then none of this would have happened.

      Except that "dn" upside-down is indistinguishable from "up". Murphy strikes again?

      --
      Accept Eris as your Fnord and personally sate her
  5. KSP by OptimalCynic · · Score: 2

    Hey, give them a break! I do that in Kerbal Space Program all the time!

    1. Re:KSP by OptimalCynic · · Score: 2

      When the rocket guides itself into the ground because you put something on upside down, it's the most appropriate analogy. I've done exactly that more than once, to the point where my mental pre-launch checklist includes "navball completely blue?"

  6. Wrong hemisphere by jovius · · Score: 4, Funny

    Should have launched from Australia.

    1. Re:Wrong hemisphere by ArcadeMan · · Score: 3, Funny

      That's the problem. Some parts were made by subcontractors in Australia, where up is down, rats are as tall as humans and hop around and flat-chested 30-years-old women are classified as underage teenagers.

  7. Shades of the US "Genesis" sample return probe... by Ellis+D.+Tripp · · Score: 4, Interesting

    which plowed into the desert floor without deploying any parachutes because a G-switch was installed backwards...

    http://www.universetoday.com/73/genesis-accident-report-released/

    --
    Remember "News for Nerds, Stuff that Matters"? Help make it a reality again! http://soylentnews.org
  8. Re:In Soviet Russia by RenderSeven · · Score: 5, Funny

    Perhaps the thinking is, as long as the arrow isnt pointed at you it's probably safe.

  9. Duck Dodgers in the 24 1/2th Century by dkleinsc · · Score: 3, Funny

    "Whoopth, I had the thilly thing in reverthe!"

    --
    I am officially gone from /. Long live http://www.soylentnews.com/
  10. I have seen something similar. by 140Mandak262Jamuna · · Score: 3, Interesting
    Back in the day when I was with the ministry of defense we lost a vehicle due to an error like this. They had changed the vendor for the gyros of the roll sensor. The new gyros had the voltages in the reverse sense. It is possible one vendor was European and the other was American. They wired it according to the sense of the old vendor. So the control input to the ailerons would add to the roll instead of counteracting it. The RPV crashed 1.5 seconds after launch.

    In the postmortem the flight director started with, "... we sadly lost the vehicle after a flight of 1.5 seconds ...". The mission director interrupted, "What flight? The damned thing had a 6000 Kg[sic][*] rocket booster. You can put it under a 3 ton rock and it will 'fly' for more than 2 seconds..."

    [*]He should have said 6000 Kgf-sec, because that was the impulse delivered by the twin rocket boosters each 1500 Kgf thrust burning for 2 seconds.

    --
    sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
  11. Mars orbital failure by Okian+Warrior · · Score: 5, Insightful

    The US once sent a probe all the way to mars, only to have it fail because the ground computer was in imperial units while the orbiter was in SI units.

    Getting everything correct is hard... really hard. For most projects you have elaborate "fail gracefully" modes which rely on external agents to notice the problem and take action. A doctor or pilot can take appropriate action, but it's hard to do with rockets.

    For comparison, I wrote the software for the altimeter that goes into some 747 aircraft. Total of about 21,000 lines of C, about 40% comments so figure 12,000 lines of code. The testers (and I) worked really hard to find all bugs in the system, knowing that a mistake could knock a plane out of the sky. There were elaborate internal checks both in software and process, and Boeing did their own testing on top of ours. Everything passed, all requirements were met, things looked good.

    The device had 1 bug, found after installation. A software typo which wasn't caught by QA even though it had a specific testing requirement. No one was negligent, it just slipped by despite best efforts.

    Multiply this by all the devices in an aircraft, and add in the other engineering disciplines like electronics and mechanical. It's really hard to get everything right all at once, and on the first try.

  12. What about the brown plume? by Antipater · · Score: 2

    I'm confused by this explanation. An upside-down angular velocity sensor would definitely pitch the rocket out of control the way it did. But what about the brown plume that was clearly visible before the rocket lost it? The consensus seemed to be that that was unburned rocket fuel, implying an engine shutdown.

    I don't build rockets, but I can't see how an upside-down rotation sensor could cause an engine shutdown, especially since the shutdown occurred before the rocket began pitching.. Could there have been more than one problem on the rocket?

    --
    Everything is better with chainsaws.
    1. Re:What about the brown plume? by Brett+Buck · · Score: 2

      I am not sure about the Proton, but earlier Russian boosters used differential throttling of the engines to control the attitude. The brown is probably excess oxidizer from running it off-mixture to throttle the engine and control the attitude - in this case to chase the erroneous gyro readings.

  13. heh by wbr1 · · Score: 4, Funny

    In Soviet Russia, snesors installed correctly, rocket installed upside down.

    --
    Silence is a state of mime.
  14. This end should point toward the ground by Arancaytar · · Score: 2

    If you want to go to space.

    If it starts pointing toward space you are having a bad problem and you will not go to space today.

  15. Sofware Fix? by bill_mcgonigle · · Score: 2

    Can the flight control system verify the sensor readings before launch? "Sensor 7 says the rocket is pointing towards the Earth on the launchpad - we might want to have a look".

    --
    My God, it's Full of Source!
    OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
  16. Just a question by 50000BTU_barbecue · · Score: 2
    Should they have launched from Australia instead?

    Yeah, weak joke, sorry.

    --
    Mostly random stuff.
  17. The short answer is "yes" by Okian+Warrior · · Score: 2

    The short answer is "yes".

    All functions range-checked their arguments on entry, calculations range-checked their results before performing further calculations, precondition logic was tested to ensure the preconditions held, periodic testing checked as many "things that should never happen" as we could think of.

    We never ignored a possibility because it was absurd, so long as there was a way to test it it was tested. The difficulty is coming up with a comprehensive list of things to check... very hard to do in practice.

  18. Contractors? by whitroth · · Score: 3, Interesting

    The Russians are using contractors, now?

    On the other hand, they seem to be doing vastly better than the US these days - we have NO WAY to put someone in orbit (unless the Pentagon's got a black program).

    We also had Challenger and Columbia. And on the latter note, I'll add that I believe my late ex's analysis, rather than the "it's falling insulation" answer. She was an engineer, and worked at the Cape for 17 years, including on the Shuttle, and she thought that some of the inspections that were supposed to be done were *not* being done, or not being done as frequently as they were supposed to have been... and that the hydraulic lines broke due to stress corrosion microcracking, and there went the aerilons.

    So, how many astronauts/cosmonauts have the Russians lost lately?

                      mark

  19. Re:Mod Parent DOWN by Brett+Buck · · Score: 2

    Not correct. If you cannot accurately sense this rate (~15 deg/hr) you will go far off the trajectory in a 10-15 minute boost. It absolutely cannot tolerate errors of a few degrees. Checking for the proper rates is an absolutely standard pre-launch check and typically, any biases in this are calibrated out while on the pad based on the known alignment of the rocket and the lat/long of the pad.

  20. Simple fix by LoRdTAW · · Score: 2

    You wouldn't need a notch, you simply move the screw holes around so they aren't square. The best method is a trapezoid pattern. Two screw holes are set closer together. Impossible to mount upside down or sideways. Or simply shift one screw hole like the ATX power supplies do.

  21. From the same people that brought you Chernobyl... by Chas · · Score: 2, Informative

    And the K19.
    And the K141 (The Kursk)
    Soyuz 1
    Soyuz 11
    And about half a dozen other fatal accidents involving shoddy workmanship.

    --


    Chas - The one, the only.
    THANK GOD!!!
  22. Professor Farnsworth ... by PPH · · Score: 2

    ... did the same thing with a seven segment timer on a bomb. He thought he had 50 minutes until it started counting seconds down "6h, 8h, Lh ...". Then Bender turned it over.

    You'd think the Russians would study other industries lessons learned and best practices.

    --
    Have gnu, will travel.