Slashdot Mirror
Upside-Down Sensors Caused Proton-M Rocket Crash
Michi writes "According to Anatoly Zak, the crash of the Russion Proton rocket on 1 July was apparently caused by several angular velocity sensors having been installed upside down. From the source: 'Each of those sensors had an arrow that was supposed to point toward the top of the vehicle, however multiple sensors on the failed rocket were pointing downward instead.' It seems amazing that something as fundamental as this was not caught during quality control. Even more amazing is that the design of the sensors permits them to be installed in the wrong orientation in the first place. Even the simplest of mechanical interlocks (such as a notch at one end that must be matched with a corresponding projection) could have prevented the accident."
A review of the quality control procedures used by the contractors responsible is underway.
...aren't so amazing when you look at the track record of Russian manufacturing.
being from there i bet half the people working on this came to work drunk and/or hung over most days
Murphy's Law is still in effect. Like the snippet says make sure that they can only be installed one way mechanically, because you won't catch 100% of the errors in QA.
Wasn't something like this responsible for the formulation of Murphy's law?
Hey, give them a break! I do that in Kerbal Space Program all the time!
Should have launched from Australia.
which plowed into the desert floor without deploying any parachutes because a G-switch was installed backwards...
http://www.universetoday.com/73/genesis-accident-report-released/
Remember "News for Nerds, Stuff that Matters"? Help make it a reality again! http://soylentnews.org
Perhaps the thinking is, as long as the arrow isnt pointed at you it's probably safe.
"Whoopth, I had the thilly thing in reverthe!"
I am officially gone from
In the postmortem the flight director started with, "... we sadly lost the vehicle after a flight of 1.5 seconds ...". The mission director interrupted, "What flight? The damned thing had a 6000 Kg[sic][*] rocket booster. You can put it under a 3 ton rock and it will 'fly' for more than 2 seconds..."
[*]He should have said 6000 Kgf-sec, because that was the impulse delivered by the twin rocket boosters each 1500 Kgf thrust burning for 2 seconds.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
We wouldn't want anything to 'appen to it...
.
Prisencolinensinainciusol. Ol Rait!
I thought the arrows pointed down because thats where the fire comes out.
The US once sent a probe all the way to mars, only to have it fail because the ground computer was in imperial units while the orbiter was in SI units.
Getting everything correct is hard... really hard. For most projects you have elaborate "fail gracefully" modes which rely on external agents to notice the problem and take action. A doctor or pilot can take appropriate action, but it's hard to do with rockets.
For comparison, I wrote the software for the altimeter that goes into some 747 aircraft. Total of about 21,000 lines of C, about 40% comments so figure 12,000 lines of code. The testers (and I) worked really hard to find all bugs in the system, knowing that a mistake could knock a plane out of the sky. There were elaborate internal checks both in software and process, and Boeing did their own testing on top of ours. Everything passed, all requirements were met, things looked good.
The device had 1 bug, found after installation. A software typo which wasn't caught by QA even though it had a specific testing requirement. No one was negligent, it just slipped by despite best efforts.
Multiply this by all the devices in an aircraft, and add in the other engineering disciplines like electronics and mechanical. It's really hard to get everything right all at once, and on the first try.
Quick! Before it's too late! Somebody call the Australian Space Agency!
Tell them to look for any boxes not marked: \/ Fragile: Then End Down \/
I'm confused by this explanation. An upside-down angular velocity sensor would definitely pitch the rocket out of control the way it did. But what about the brown plume that was clearly visible before the rocket lost it? The consensus seemed to be that that was unburned rocket fuel, implying an engine shutdown.
I don't build rockets, but I can't see how an upside-down rotation sensor could cause an engine shutdown, especially since the shutdown occurred before the rocket began pitching.. Could there have been more than one problem on the rocket?
Everything is better with chainsaws.
Gorbachev?
In Soviet Russia, snesors installed correctly, rocket installed upside down.
Silence is a state of mime.
It wasn't that they were installed upside down, they were installed for a southern hemisphere launch!
Interesting article on Space News
http://www.spacenews.com/article/launch-report/36112proton-launch-failures-more-likely-when-russia-footing-the-bill#.Ud2DnPkyZ8E/
that points out that the Proton launch failures have a mysterious correlation to whether the customer is private or government (with government launches being the unlucky ones).
----------------------------------- My Other Sig Is Hilarious -----------------------------------
If you want to go to space.
If it starts pointing toward space you are having a bad problem and you will not go to space today.
I am confused - did the upside-down sensors cause the other problems as well, such as the early disconnect of wiring, or are these all separate failures? If it's the latter, there needs to be some serious effort made to improve the design and construction.
We just assume that the sensors were upside down -- but does anyone ask if the rocker wasn't upside down and the sensors right side up?
No. No they do not. Installing sensors is a thankless job and nobody says; "Great sensor." They only talk to you if something goes wrong."
>> Brought to you by the Anti Sensor Installer Defamation League
>>"ad space available -- low rates!!!"
What stops the key from being installed wrongly?
The design of the key and the tooling and processes used to produce it. Speaking generally you use behavior shaping constraints which prevent incorrect assembly. Proper design, interlocks, jigs and fixtures, automated tooling, and lots of other tools are used to eliminate mistakes.
Anything that relies on visual inspection by a human WILL eventually have an error. My company makes wire harnesses and every time we are forced to rely on a visual inspection process there inevitably are some errors. Most of the time the need for these visual inspection can be done away with with product design and in some cases some tooling. However many engineers can't be bothered to design for assembly or the cost of the mistake proofing is not justified by cost of an error.
Can the flight control system verify the sensor readings before launch? "Sensor 7 says the rocket is pointing towards the Earth on the launchpad - we might want to have a look".
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
Even the simplest of mechanical interlocks (such as a notch at one end that must be matched with a corresponding projection)
This only moves the problem, it doesn't fix it. There is now the possibility for the sensors to be installed correctly into mechanical interlocks that were themselves installed upside down
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
Yeah, weak joke, sorry.
Mostly random stuff.
In 2011 the Mars-bound Russian Fobos-Grunt failed because of a programming error which led to a simultaneous reboot of two working channels of an onboard computer, leaving the craft parked in low-Earth orbit and eventually uncontrolled re-entry over the Pacific Ocean. In 1999 NASA's Mars Climate Orbiter disintegrated in the Martian upper atmosphere due to ground based computer software which produced output in non-SI units of pound-seconds (lbf×s) instead of the metric units of newton-seconds (N×s) specified in the contract between NASA and Lockheed.
When you try to make things cheaper you get failures.
Do not look at laser with remaining good eye.
things are always unstable during test periods. once a device this complicated, like a space booster or a 787 for instance, gets certified and enters serial production, that is the part where inspections and workers empowered to shut down the line becomes the paramount safety mechanism.
if this is supposed to be a new economy, how come they still want my old fashioned money?
This brings to mind a project that I was involved with. part of the system involved an electronic compass that provided bearing information to a computer. On testing, someone noticed that the bearing angle seemed to be reversed A programmer cleverly fixed the "problem" in software - obviously it must have been some kind of bug in the compass software. What had actually happened was that the compass had been installed upside down and the software fix was not completely successful - the compass had a tilt compensation feature that worked incorrectly in this configuration.
"For the want of a pennyworth of tar, the ship was lost."
We have sayings covering this system failure going back centuries, and probably even thousands of years. In London for instance, a few years make, a fuel depot near London was lost at the cost of hundreds of millions of pounds, because they would not emply a single night-watchman- an employee who would have spotted the small, clearly visible, external leak.
Why is it that even when a system is a multi-billion dollar one, there is penny-scraping at the bottom where mission critical work is being done.
You know what makes a difference? NASA producing building full of written regulations defining everything? NO! Paying the people in charge salaries of hundreds of millions of dollars? NO! Creating a regime where everyone wants to take pride in their work, and everyone feels their opinions can make a difference? YES!
Do you seriosly thing that the people handling those sensors were NOT amazed and disgusted that they lacked fool-proof orientation mechanisms? Why then was this situation not corrected? Well, here's an analogue. Those useless cretins at Intel are responible for every modern interconnect used on hundreds of millions of PCs. Notice that USB plugs lack a positive orientation, and that you have to 'try' inserting to discover which way round they go. (BTW, USB is also useless at the electrical AND network layers too). Did Intel make these angular velocity sensors (I joke, but not really)?
"They'll know what we mean" is the cry of every useless engineer. Here's the test. An Intel class engineer will place the words "open this end" on a box. A decent engineer will place the words "open the OTHER end" on the same box. THINK ABOUT IT!
Anyway, anyone with a shred of intelligence in this business KNOWS all critical parts/plugs must be given a keyed receptacle that can ONLY take the plug/part in the correct orientation. In the case of a plug (linear with lots of connectors) that could be forced to fit in the wrong orientation (and Humans always find a way to do this), it is essential that no proper fuctioned appears to occur, and if possible no damage is done to the system (through power lines transposed etc).
This angular velocity sensor shouldn't have even tested correctly when fitted the wrong way round. Have these clowns never heard of diodes and the like.
Being old enough to have done 'IT' when IT referenced old punch-card class computing, I vividly recall hearing about the specialist job of 'data entry', and how all data was entered by TWO seperate people, allowing the correctness to be cross-checked. Of course someone checked the work that built this rocket, BUT with the purpose of signing off on the work at the lowest possible cost.
Done properly the checker would be comparing a photo of the ideal build with the actual build, but the sensors may have been 'new' parts replacing the previously used ones, and thus looked different. Indeed, if new sensors were being used, it is possible they replaced previous designs that did have positive but 'different' positive orientation insertion mechanisms. Anyone who builds anything knows the issue of having to swap the parts being used at short-notice. The new part may (hopefully on a temp build basis) have to be 'forced' to fit in the location where the old part went.
One might argue that the Proton rocket builders should havekept massive stocks of 'spare' parts, BUT this ignores several factors. Some parts may 'rot' in storage (perhaps a new alloy is suffering fatal electo-migration after a few years). Some parts of the design may require an upgrade after investigations prove the existing design is inadequate in some respect.However, this does not mitigate the care with which new parts must be introduced. The 'build' guidelines must be updated all along the production line. New photographs of correct builds must be made, and given to workers/inspectors. Foolproof orientation mechanisms must be introduced as soon as possible for new parts (although this may take
The short answer is "yes".
All functions range-checked their arguments on entry, calculations range-checked their results before performing further calculations, precondition logic was tested to ensure the preconditions held, periodic testing checked as many "things that should never happen" as we could think of.
We never ignored a possibility because it was absurd, so long as there was a way to test it it was tested. The difficulty is coming up with a comprehensive list of things to check... very hard to do in practice.
The Thor missile, the first of the intermediate range ballistic missiles had the same issue on its 2nd launch causing the range operator to hit the self destruct button in fear that it was going the wrong way.
"Where is my mind?"
'a' G-switch ?
*all* of them were installed upside down. (although 'all' might have only been two)
Build it, and they will come^Hplain.
The loud sound of palms slapping foreheads after that was discovered caused a cow to give birth.
The Russians are using contractors, now?
On the other hand, they seem to be doing vastly better than the US these days - we have NO WAY to put someone in orbit (unless the Pentagon's got a black program).
We also had Challenger and Columbia. And on the latter note, I'll add that I believe my late ex's analysis, rather than the "it's falling insulation" answer. She was an engineer, and worked at the Cape for 17 years, including on the Shuttle, and she thought that some of the inspections that were supposed to be done were *not* being done, or not being done as frequently as they were supposed to have been... and that the hydraulic lines broke due to stress corrosion microcracking, and there went the aerilons.
So, how many astronauts/cosmonauts have the Russians lost lately?
mark
What you do mean there is no angular velocity? It's on the Earth and the Earth is turning. Gyros for this application can easily sense Earth rate, that's an absolutely classic check for proper operation prelaunch.
if it's something critical and only functions correctly in one direction, you need to design it so that it only fits one way.
and it's not just a Beastie Boys song anymore
Closed loop design is not a part of the Russian way, apparently.
Trust me when I say that is not remotely unique to the Russians. You wouldn't believe how poorly designed some of the products I deal with are. Virtually every drawing we get requires some amount of redesign because it either can't be built or will work badly. We had a customer that made heart lung machines who didn't even have an accurate bill of materials. I'm not talking just a little bit off either - they had NO idea what most of the part numbers were. We had to tell them the parts that were in their own product.
I've never considered this a good way to denote orientation - you have to learn that the arrow is supposed to point up. If you ask a random person on the street who's never seen this (it's common in packaging) which way the arrow is supposed to point, I'll bet half will say down because that's the way gravity pulls things.
A mechanical interlock is one of those things which is unappreciated when it's there. So what happens is you design stuff with interlocks for years, and nothing goes wrong. Then someone thinks, "Nothing has ever gone wrong with this before, why bother designing a safety system to prevent a problem which has never occurred?" They skip the interlock this iteration, and it causes a catastrophic failure.
"Even the simplest of mechanical interlocks (such as a notch at one end that must be matched with a corresponding projection) could have prevented the accident."
Unless the sockets are installed upside-down.....
love is just extroverted narcissism
True Communism works perfectly. Until people are involved.
Marx could never quite figure out that little detail.
NASA's "Stardust" sample return that used aerogel to capture solar wind and solar system dust crash-landed in the Utah desert because of a similar sensor installed in the opposite orientation from where it should have been in the re-entry vehicle.
There are no velocity sensors that work on rockets. There are however acceleration sensors. These can quite handily be tested on the ground using the 1G acceleration equivalent gravitational pull. It is a simple test, takes seconds, and immediately shows orientation very precisely. Not testing orientation on these sensors after assembly is exceedingly stupid.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
I was actually in the audience when one of the guys involved in this project (an engineer, but high up) was giving a presentation discussing the event. It's not as simple as "some moron put this in backwards!!!" like everyone here seems to think. It was actually because the design indicated the sensor should be installed in the wrong orientation and no one caught it. It went past multiple reviews and was observed by tens, if not hundreds, of pairs of eyes. There was a test that they could have performed that would have caught the problem, but the decision was made to skip the test because the project was running behind and they had to cut out some of the tests. Multiple people signed off on that decision. Eventually it was determined that the disaster was no one person's fault and that it was simply something that managed to slip by everyone.
You also have to realize that this was one part out of a million. People are claiming that everyone involved is a complete dumbass or entirely incompetent because they screwed up a single part out of all the parts that make up a spacecraft.
would have saved the rocket.
Its fucking the rocket science industry.
I mean I cannot perceive how this was not caught. At some point while the rocket is sitting on the launch pad, someone must have run through some kind of test diagnostics, any sensor involved in resolving direction should be on that list of checks:
Sensor 1 Direction: UP - OK
Sensor 2 Direction: UP - OK
Sensor 3 Direction: UP - OK
Sensor 4 Direction: DOWN - Abort!, Abort!, Abort!
Anyways, its easy to over trivialize this as we don't know what is really involved in making a rocket go...oh yes, its to make sure fire comes out the bottom and the rest of it goes UP!
I haven't thought of anything clever to put here, but then again most of you haven't either.
None. It wasn't any scientist. The real story.
JPL did the previous mission themselves (Pathfinder), and JPL like all scientists used metric. A later mission, a congressdroid insists that some of it gets outsourced to one of the military-aerospace contractors with a plant in his district. This contractor has an old geezer running the division who insists only on imperial.
So JPL and Hockleed have to go back and forth for their navigation procedures. Contractor puts a fresh college graduate on the program, you know to lower costs.
Which is why airplanes still have multiple, independently-developed systems installed despite all of the prior checks and controls.
It's not a practical solution for rockets though.
You are correct, and IMO this is the right way to do things.
Note that 747's have two altimeters running the same software. It was pointed out in development that if one of them had a problem, the other one would likely exhibit the same problem at the same time.
This is a subject close to my heart, and which deserves a lot of thought and discussion. We're putting lots of software into medical and aircraft these days. Software in cars does not get the same level of scrutiny or regulatory process, and with the advent of self-driving cars perhaps they should.
I'd be interested in people's opinions on this. Where should we draw the line on regulatory process? Aircraft and medical are obvious, but how about cars, smart [power] meters, phone COs, or industrial controllers? Should the West Texas fertilizer plant have had regulatory oversight on their control systems, for instance?
Having a program crash the user's PC is relatively benign and can be handled as a customer service issue. Nowadays we're putting software in many more places which affects public safety.
Maybe Bill the cat had a new job.. while in Russia.
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
"First things first -- but not necessarily in that order"
-- The Doctor, "Doctor
They should have launched from Australia. That would solve the arrow upside-down problem.
You wouldn't need a notch, you simply move the screw holes around so they aren't square. The best method is a trapezoid pattern. Two screw holes are set closer together. Impossible to mount upside down or sideways. Or simply shift one screw hole like the ATX power supplies do.
Well, in an attempt to get things back on topic... I recall one of the big differences in the US and Russian space programs was that the US would build their rockets upright and roll them out to the launch pad that way, while the Russians would build their rockets sideways, and then hoist them upright at the launchpad.
So really it's not an issue of them not knowing up from down, but left from right.
OK, now go back into reading that politically what you will.
And the K19.
And the K141 (The Kursk)
Soyuz 1
Soyuz 11
And about half a dozen other fatal accidents involving shoddy workmanship.
Chas - The one, the only.
THANK GOD!!!
Video of the take off and crash.
Jhyrryl
You'd think the Russians would study other industries lessons learned and best practices.
Have gnu, will travel.
Ah yes. Simplified English.
Have gnu, will travel.
Is this any different than confusing inches and centimeters?
In fairness, some of those (Chernobyl, certainly) were the result of flawed _design_, not workmanship.
In Soviet Russia, they are called conscripts and not contractors. Work is difficult, often with complaints of "This thing is heavy" but after awhile, they followed them up with praise like "At least we have job!" & "For Mother Russia!" so it all turned out well in end.
Another fine example of the Russian Reversal.
Have gnu, will travel.
Lev Andropov: It's stuck, yes?
Watts: Back off! You don't know the components!
Lev Andropov: [annoyed] Components. American components, Russian Components, ALL MADE IN TAIWAN!
One from my history : oil production platform was having an add-on applied to it - a module that would compress produced gas and pump it down a newly installed pipeline for sale onshore, instead of being used for fuel and burned off. Project required some machinery that connected to the existing machinery on the platform, plus a steel jacket set on the seabed to support the new machinery.
Long story short : left hand design team built the support jacket on the assumption that sea level was figure X above seabed (a measure called "Lowest Astronomical Tides") ; right hand design team designed the surface equipment, pumps, etc on the assumption that things were so far above sea-level. But they used a measure of sea level called "Mean Sea Level".
Major fuck up ; things didn't fit ; lines that should have let liquids run back to the platform sloped the wrong way ; no room or weight to put pumps onto the lines. M.A.J.O.R. fuck up. Took months to bodge up a solution. People responsible either fired or promoted - I never found out which. Situation 'Normal' : All Fucked Up.
Birds are not dinosaur descendants;birds are dinosaurs, for all useful meanings of "birds", "are" and "dinosaurs"