Slashdot Mirror
Upside-Down Sensors Caused Proton-M Rocket Crash
Michi writes "According to Anatoly Zak, the crash of the Russion Proton rocket on 1 July was apparently caused by several angular velocity sensors having been installed upside down. From the source: 'Each of those sensors had an arrow that was supposed to point toward the top of the vehicle, however multiple sensors on the failed rocket were pointing downward instead.' It seems amazing that something as fundamental as this was not caught during quality control. Even more amazing is that the design of the sensors permits them to be installed in the wrong orientation in the first place. Even the simplest of mechanical interlocks (such as a notch at one end that must be matched with a corresponding projection) could have prevented the accident."
A review of the quality control procedures used by the contractors responsible is underway.
...aren't so amazing when you look at the track record of Russian manufacturing.
blow up moose and squirrel....
sounds like some engineers are going to be working the siberian salt mines.
Upside down is not always wrong.
being from there i bet half the people working on this came to work drunk and/or hung over most days
Murphy's Law is still in effect. Like the snippet says make sure that they can only be installed one way mechanically, because you won't catch 100% of the errors in QA.
Wasn't something like this responsible for the formulation of Murphy's law?
Hey, give them a break! I do that in Kerbal Space Program all the time!
Garbage in, garbage out.
Vodka may have been involved.
Should have launched from Australia.
Were they shipped in a cardboard box marked with an arrow and "this end up"? That would explain why. Nobody pays attention to that.
"It seems amazing that something as fundamental as this was not caught during quality control"
I have an explanation: it's Russian
which plowed into the desert floor without deploying any parachutes because a G-switch was installed backwards...
http://www.universetoday.com/73/genesis-accident-report-released/
Remember "News for Nerds, Stuff that Matters"? Help make it a reality again! http://soylentnews.org
"Whoopth, I had the thilly thing in reverthe!"
I am officially gone from
In the postmortem the flight director started with, "... we sadly lost the vehicle after a flight of 1.5 seconds ...". The mission director interrupted, "What flight? The damned thing had a 6000 Kg[sic][*] rocket booster. You can put it under a 3 ton rock and it will 'fly' for more than 2 seconds..."
[*]He should have said 6000 Kgf-sec, because that was the impulse delivered by the twin rocket boosters each 1500 Kgf thrust burning for 2 seconds.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
. . . In Russian?
We wouldn't want anything to 'appen to it...
.
Prisencolinensinainciusol. Ol Rait!
I thought the arrows pointed down because thats where the fire comes out.
The US once sent a probe all the way to mars, only to have it fail because the ground computer was in imperial units while the orbiter was in SI units.
Getting everything correct is hard... really hard. For most projects you have elaborate "fail gracefully" modes which rely on external agents to notice the problem and take action. A doctor or pilot can take appropriate action, but it's hard to do with rockets.
For comparison, I wrote the software for the altimeter that goes into some 747 aircraft. Total of about 21,000 lines of C, about 40% comments so figure 12,000 lines of code. The testers (and I) worked really hard to find all bugs in the system, knowing that a mistake could knock a plane out of the sky. There were elaborate internal checks both in software and process, and Boeing did their own testing on top of ours. Everything passed, all requirements were met, things looked good.
The device had 1 bug, found after installation. A software typo which wasn't caught by QA even though it had a specific testing requirement. No one was negligent, it just slipped by despite best efforts.
Multiply this by all the devices in an aircraft, and add in the other engineering disciplines like electronics and mechanical. It's really hard to get everything right all at once, and on the first try.
So are the responsible going to jail... just like programmers go for bad development?
Quick! Before it's too late! Somebody call the Australian Space Agency!
Tell them to look for any boxes not marked: \/ Fragile: Then End Down \/
I'm confused by this explanation. An upside-down angular velocity sensor would definitely pitch the rocket out of control the way it did. But what about the brown plume that was clearly visible before the rocket lost it? The consensus seemed to be that that was unburned rocket fuel, implying an engine shutdown.
I don't build rockets, but I can't see how an upside-down rotation sensor could cause an engine shutdown, especially since the shutdown occurred before the rocket began pitching.. Could there have been more than one problem on the rocket?
Everything is better with chainsaws.
> Each of those sensors had an arrow that was supposed to point toward the top of the vehicle, however multiple sensors on the failed rocket were pointing downward instead.
The same reason brought down the soviet "Polyus" battle-satellite in 1987, as it was trying to ride into LEO, piggybacking a giant Energija rocket booster.
In Soviet Russia, snesors installed correctly, rocket installed upside down.
Silence is a state of mime.
It wasn't that they were installed upside down, they were installed for a southern hemisphere launch!
Of course it's possible to add more controls and archive more security, but it also adds weight of vessel which in turn it cannot raise to orbit. Even if its a small thing those easily adds up. That's my first thought of this event.
Anyway, this brings in to my mind a story loooong time back from -70's when I was still quite young. My father was a paper machinery pioneer and the company he was employed had sold a paper mill to Russia and was about to be delivered to city of Archangel (Koala Peninsula). The project was quite in the last moments some summer and we were spending summer holiday at our summer cottage when a company car was suddenly drove there. The driver had just a note telling that my father had to return immediately to work. There was some kind of severe issue with the machinery that had arrived to plant. Without further ado my father shaved his holiday beard and kissed us, her wife and me his son around 12 then, goodbye. He just quipped that he could be back by the evening or possibly next day. Well the days passed and we heard nothing. After 4 days we took a bus back to town were we lived and went to home. There was a short notice on kitchen table that he had to leave for a short trip to USSR to find out what are the problems with installation of the machinery.
We heard nothing from him nor did his employer. He returned after 3 1/2 weeks and looked like he had lived in ghetto, he had lost weight, hadn't shaved, clothes were dirty and he smelled like a rat. Right after he had been in sauna and slept well, he next day told what had happened. He told that Russian customer (state committee or ministry which purchased all this large projects) had demanded that they send 3 to engineers (one of these my father) and their supervisor immediately to the site to solve the issues they had with the installation. The company (Valmet at that time) even though it was state owned too had agreed because they considered customer so important. So they drove with that same Lincoln from Central Finland towards Leningrad (St. Petersburgh) picking up Russian interpreter on their way and continued immediately to Archangel. To make the story shorter I just refer that he said it was a journey he never forget so many things happened next 12 hour while driving. OK, so they got to newly built paper mill site and went directly to see the problem. They found out that the site was completely built by the prisoners and they could only enter the site while there were no prisoners nor their guards or supervisors.
Right, they found out quite soon that in that installation group were none, I repeat none, who could have been able to read technical documentation and understand it in details so that they could have done the preparations needed properly. The problem was that they had prepared the whole concrete base of the machinery so that it was all reversed left to right. They went and tried to explain the issue using interpreter to plant managers who seemed not first to understand the extent of the problem at all. To understand the scale think of it a paper mill of that time was 7-9 meters wide, it was about 400-500 meters long. Fixing the problem would take months as they have to first break and dig out the old, make new molds and then cast new concrete, wait it dries up enough and then they would be able to start proper installation. Once they got the word trough and installation and plant managers understand the scale they started demanding that the company had provided wrong machinery which left and right was mismatched and because of that the company had to deliver a completely new machinery etc.
OK, just to shorten the long story more it's enough that I just tell that my father and his team spent several days before they got everybody convinced and that installation was not possible before they fix the base of the machinery properly. Knowing that would take very long and that they could come back once it's finished Russians suddenly didn't agree them to leave before the installation was done. They got the wo
Interesting article on Space News
http://www.spacenews.com/article/launch-report/36112proton-launch-failures-more-likely-when-russia-footing-the-bill#.Ud2DnPkyZ8E/
that points out that the Proton launch failures have a mysterious correlation to whether the customer is private or government (with government launches being the unlucky ones).
----------------------------------- My Other Sig Is Hilarious -----------------------------------
If you want to go to space.
If it starts pointing toward space you are having a bad problem and you will not go to space today.
I am confused - did the upside-down sensors cause the other problems as well, such as the early disconnect of wiring, or are these all separate failures? If it's the latter, there needs to be some serious effort made to improve the design and construction.
We just assume that the sensors were upside down -- but does anyone ask if the rocker wasn't upside down and the sensors right side up?
No. No they do not. Installing sensors is a thankless job and nobody says; "Great sensor." They only talk to you if something goes wrong."
>> Brought to you by the Anti Sensor Installer Defamation League
>>"ad space available -- low rates!!!"
The components were well engineered, but the design of the component was lacking. Installing something backwards should not be possible, and that's part of the design too. Installing computer memory chips is really difficult to do if you install them backwards, because there is a notch preventing it. I've built printed circuit boards (professionally) and have also installed motherboards into computers. In most cases you can't install the wiring harnesses incorrectly because either it doesn't reach the socket, or the number of pins is incorrect. Installing microprocessor chips backwards is impossible because of the notch, and the pins would be up (not able to reach the ball grid array at the bottom) if you put it in upside down.
What stops the key from being installed wrongly?
The design of the key and the tooling and processes used to produce it. Speaking generally you use behavior shaping constraints which prevent incorrect assembly. Proper design, interlocks, jigs and fixtures, automated tooling, and lots of other tools are used to eliminate mistakes.
Anything that relies on visual inspection by a human WILL eventually have an error. My company makes wire harnesses and every time we are forced to rely on a visual inspection process there inevitably are some errors. Most of the time the need for these visual inspection can be done away with with product design and in some cases some tooling. However many engineers can't be bothered to design for assembly or the cost of the mistake proofing is not justified by cost of an error.
Can the flight control system verify the sensor readings before launch? "Sensor 7 says the rocket is pointing towards the Earth on the launchpad - we might want to have a look".
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
Even the simplest of mechanical interlocks (such as a notch at one end that must be matched with a corresponding projection)
This only moves the problem, it doesn't fix it. There is now the possibility for the sensors to be installed correctly into mechanical interlocks that were themselves installed upside down
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
Yeah, weak joke, sorry.
Mostly random stuff.
In 2011 the Mars-bound Russian Fobos-Grunt failed because of a programming error which led to a simultaneous reboot of two working channels of an onboard computer, leaving the craft parked in low-Earth orbit and eventually uncontrolled re-entry over the Pacific Ocean. In 1999 NASA's Mars Climate Orbiter disintegrated in the Martian upper atmosphere due to ground based computer software which produced output in non-SI units of pound-seconds (lbf×s) instead of the metric units of newton-seconds (N×s) specified in the contract between NASA and Lockheed.
When you try to make things cheaper you get failures.
Do not look at laser with remaining good eye.
I remember seeing a box in a warehouse that had "This end not up!" printed on the side.
Good idea!
things are always unstable during test periods. once a device this complicated, like a space booster or a 787 for instance, gets certified and enters serial production, that is the part where inspections and workers empowered to shut down the line becomes the paramount safety mechanism.
if this is supposed to be a new economy, how come they still want my old fashioned money?
You blow up rockets.
I could imagine that some modules of a rocket could be assembled in a 'downward' facing orientation.
This brings to mind a project that I was involved with. part of the system involved an electronic compass that provided bearing information to a computer. On testing, someone noticed that the bearing angle seemed to be reversed A programmer cleverly fixed the "problem" in software - obviously it must have been some kind of bug in the compass software. What had actually happened was that the compass had been installed upside down and the software fix was not completely successful - the compass had a tilt compensation feature that worked incorrectly in this configuration.
It's somehow satisfying to know that contractors overbilling the government, stealing from taxpayers, and generally being inept while promoting the "efficiency" of contracting out government work is apparently how it works everywhere, even Russia.
"For the want of a pennyworth of tar, the ship was lost."
We have sayings covering this system failure going back centuries, and probably even thousands of years. In London for instance, a few years make, a fuel depot near London was lost at the cost of hundreds of millions of pounds, because they would not emply a single night-watchman- an employee who would have spotted the small, clearly visible, external leak.
Why is it that even when a system is a multi-billion dollar one, there is penny-scraping at the bottom where mission critical work is being done.
You know what makes a difference? NASA producing building full of written regulations defining everything? NO! Paying the people in charge salaries of hundreds of millions of dollars? NO! Creating a regime where everyone wants to take pride in their work, and everyone feels their opinions can make a difference? YES!
Do you seriosly thing that the people handling those sensors were NOT amazed and disgusted that they lacked fool-proof orientation mechanisms? Why then was this situation not corrected? Well, here's an analogue. Those useless cretins at Intel are responible for every modern interconnect used on hundreds of millions of PCs. Notice that USB plugs lack a positive orientation, and that you have to 'try' inserting to discover which way round they go. (BTW, USB is also useless at the electrical AND network layers too). Did Intel make these angular velocity sensors (I joke, but not really)?
"They'll know what we mean" is the cry of every useless engineer. Here's the test. An Intel class engineer will place the words "open this end" on a box. A decent engineer will place the words "open the OTHER end" on the same box. THINK ABOUT IT!
Anyway, anyone with a shred of intelligence in this business KNOWS all critical parts/plugs must be given a keyed receptacle that can ONLY take the plug/part in the correct orientation. In the case of a plug (linear with lots of connectors) that could be forced to fit in the wrong orientation (and Humans always find a way to do this), it is essential that no proper fuctioned appears to occur, and if possible no damage is done to the system (through power lines transposed etc).
This angular velocity sensor shouldn't have even tested correctly when fitted the wrong way round. Have these clowns never heard of diodes and the like.
Being old enough to have done 'IT' when IT referenced old punch-card class computing, I vividly recall hearing about the specialist job of 'data entry', and how all data was entered by TWO seperate people, allowing the correctness to be cross-checked. Of course someone checked the work that built this rocket, BUT with the purpose of signing off on the work at the lowest possible cost.
Done properly the checker would be comparing a photo of the ideal build with the actual build, but the sensors may have been 'new' parts replacing the previously used ones, and thus looked different. Indeed, if new sensors were being used, it is possible they replaced previous designs that did have positive but 'different' positive orientation insertion mechanisms. Anyone who builds anything knows the issue of having to swap the parts being used at short-notice. The new part may (hopefully on a temp build basis) have to be 'forced' to fit in the location where the old part went.
One might argue that the Proton rocket builders should havekept massive stocks of 'spare' parts, BUT this ignores several factors. Some parts may 'rot' in storage (perhaps a new alloy is suffering fatal electo-migration after a few years). Some parts of the design may require an upgrade after investigations prove the existing design is inadequate in some respect.However, this does not mitigate the care with which new parts must be introduced. The 'build' guidelines must be updated all along the production line. New photographs of correct builds must be made, and given to workers/inspectors. Foolproof orientation mechanisms must be introduced as soon as possible for new parts (although this may take
The short answer is "yes".
All functions range-checked their arguments on entry, calculations range-checked their results before performing further calculations, precondition logic was tested to ensure the preconditions held, periodic testing checked as many "things that should never happen" as we could think of.
We never ignored a possibility because it was absurd, so long as there was a way to test it it was tested. The difficulty is coming up with a comprehensive list of things to check... very hard to do in practice.
The Thor missile, the first of the intermediate range ballistic missiles had the same issue on its 2nd launch causing the range operator to hit the self destruct button in fear that it was going the wrong way.
"Where is my mind?"
'a' G-switch ?
*all* of them were installed upside down. (although 'all' might have only been two)
Build it, and they will come^Hplain.
The loud sound of palms slapping foreheads after that was discovered caused a cow to give birth.
The Russians are using contractors, now?
On the other hand, they seem to be doing vastly better than the US these days - we have NO WAY to put someone in orbit (unless the Pentagon's got a black program).
We also had Challenger and Columbia. And on the latter note, I'll add that I believe my late ex's analysis, rather than the "it's falling insulation" answer. She was an engineer, and worked at the Cape for 17 years, including on the Shuttle, and she thought that some of the inspections that were supposed to be done were *not* being done, or not being done as frequently as they were supposed to have been... and that the hydraulic lines broke due to stress corrosion microcracking, and there went the aerilons.
So, how many astronauts/cosmonauts have the Russians lost lately?
mark
What you do mean there is no angular velocity? It's on the Earth and the Earth is turning. Gyros for this application can easily sense Earth rate, that's an absolutely classic check for proper operation prelaunch.
if it's something critical and only functions correctly in one direction, you need to design it so that it only fits one way.
and it's not just a Beastie Boys song anymore
Closed loop design is not a part of the Russian way, apparently.
Trust me when I say that is not remotely unique to the Russians. You wouldn't believe how poorly designed some of the products I deal with are. Virtually every drawing we get requires some amount of redesign because it either can't be built or will work badly. We had a customer that made heart lung machines who didn't even have an accurate bill of materials. I'm not talking just a little bit off either - they had NO idea what most of the part numbers were. We had to tell them the parts that were in their own product.
I've never considered this a good way to denote orientation - you have to learn that the arrow is supposed to point up. If you ask a random person on the street who's never seen this (it's common in packaging) which way the arrow is supposed to point, I'll bet half will say down because that's the way gravity pulls things.
A mechanical interlock is one of those things which is unappreciated when it's there. So what happens is you design stuff with interlocks for years, and nothing goes wrong. Then someone thinks, "Nothing has ever gone wrong with this before, why bother designing a safety system to prevent a problem which has never occurred?" They skip the interlock this iteration, and it causes a catastrophic failure.
"Even the simplest of mechanical interlocks (such as a notch at one end that must be matched with a corresponding projection) could have prevented the accident."
Unless the sockets are installed upside-down.....
love is just extroverted narcissism
True Communism works perfectly. Until people are involved.
Marx could never quite figure out that little detail.
NASA's "Stardust" sample return that used aerogel to capture solar wind and solar system dust crash-landed in the Utah desert because of a similar sensor installed in the opposite orientation from where it should have been in the re-entry vehicle.
There are no velocity sensors that work on rockets. There are however acceleration sensors. These can quite handily be tested on the ground using the 1G acceleration equivalent gravitational pull. It is a simple test, takes seconds, and immediately shows orientation very precisely. Not testing orientation on these sensors after assembly is exceedingly stupid.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
I was actually in the audience when one of the guys involved in this project (an engineer, but high up) was giving a presentation discussing the event. It's not as simple as "some moron put this in backwards!!!" like everyone here seems to think. It was actually because the design indicated the sensor should be installed in the wrong orientation and no one caught it. It went past multiple reviews and was observed by tens, if not hundreds, of pairs of eyes. There was a test that they could have performed that would have caught the problem, but the decision was made to skip the test because the project was running behind and they had to cut out some of the tests. Multiple people signed off on that decision. Eventually it was determined that the disaster was no one person's fault and that it was simply something that managed to slip by everyone.
You also have to realize that this was one part out of a million. People are claiming that everyone involved is a complete dumbass or entirely incompetent because they screwed up a single part out of all the parts that make up a spacecraft.
would have saved the rocket.
Its fucking the rocket science industry.
I mean I cannot perceive how this was not caught. At some point while the rocket is sitting on the launch pad, someone must have run through some kind of test diagnostics, any sensor involved in resolving direction should be on that list of checks:
Sensor 1 Direction: UP - OK
Sensor 2 Direction: UP - OK
Sensor 3 Direction: UP - OK
Sensor 4 Direction: DOWN - Abort!, Abort!, Abort!
Anyways, its easy to over trivialize this as we don't know what is really involved in making a rocket go...oh yes, its to make sure fire comes out the bottom and the rest of it goes UP!
I haven't thought of anything clever to put here, but then again most of you haven't either.
None. It wasn't any scientist. The real story.
JPL did the previous mission themselves (Pathfinder), and JPL like all scientists used metric. A later mission, a congressdroid insists that some of it gets outsourced to one of the military-aerospace contractors with a plant in his district. This contractor has an old geezer running the division who insists only on imperial.
So JPL and Hockleed have to go back and forth for their navigation procedures. Contractor puts a fresh college graduate on the program, you know to lower costs.
It is the best quote from obligatory xkcd number 1133: The Up Goer Five
A diagram of the Saturn Five rocket only using the top ten thousand words people use the most often.
Which is why airplanes still have multiple, independently-developed systems installed despite all of the prior checks and controls.
It's not a practical solution for rockets though.
You are correct, and IMO this is the right way to do things.
Note that 747's have two altimeters running the same software. It was pointed out in development that if one of them had a problem, the other one would likely exhibit the same problem at the same time.
This is a subject close to my heart, and which deserves a lot of thought and discussion. We're putting lots of software into medical and aircraft these days. Software in cars does not get the same level of scrutiny or regulatory process, and with the advent of self-driving cars perhaps they should.
I'd be interested in people's opinions on this. Where should we draw the line on regulatory process? Aircraft and medical are obvious, but how about cars, smart [power] meters, phone COs, or industrial controllers? Should the West Texas fertilizer plant have had regulatory oversight on their control systems, for instance?
Having a program crash the user's PC is relatively benign and can be handled as a customer service issue. Nowadays we're putting software in many more places which affects public safety.
Maybe Bill the cat had a new job.. while in Russia.
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
"First things first -- but not necessarily in that order"
-- The Doctor, "Doctor
The Russkie talks big, but frankly, we think he's short of know how. I mean, you just can't expect a bunch of ignorant peons to understand a machine like some of our boys. And that's not meant as an insult, Mr. Ambassador, I mean, you take your average Russkie, we all know how much guts he's got. Hell, lookit look at all them them Nazis killed off and they still wouldn't quit.
This is what happens when Russia lets businessmen handle their space efforts.
They should have launched from Australia. That would solve the arrow upside-down problem.
You wouldn't need a notch, you simply move the screw holes around so they aren't square. The best method is a trapezoid pattern. Two screw holes are set closer together. Impossible to mount upside down or sideways. Or simply shift one screw hole like the ATX power supplies do.
Well, in an attempt to get things back on topic... I recall one of the big differences in the US and Russian space programs was that the US would build their rockets upright and roll them out to the launch pad that way, while the Russians would build their rockets sideways, and then hoist them upright at the launchpad.
So really it's not an issue of them not knowing up from down, but left from right.
OK, now go back into reading that politically what you will.
And the K19.
And the K141 (The Kursk)
Soyuz 1
Soyuz 11
And about half a dozen other fatal accidents involving shoddy workmanship.
Chas - The one, the only.
THANK GOD!!!
Video of the take off and crash.
Jhyrryl
You'd think the Russians would study other industries lessons learned and best practices.
Have gnu, will travel.
Is this any different than confusing inches and centimeters?
Why would you make an interlock to prevent them from being installed backwards? There could be many instances where you want to install them backwards... but some damn engineer designed them not to be!
In fairness, some of those (Chernobyl, certainly) were the result of flawed _design_, not workmanship.
Enough about screw hole patterns. Rockets aren't designed to carry extra weight or bulk without reason, so cables are are trimmed to fit. This sensor seems to have a cable coming off from it. How was it possible for the cable to be connected? Why was the cable long enough to allow this?
In Soviet Russia, they are called conscripts and not contractors. Work is difficult, often with complaints of "This thing is heavy" but after awhile, they followed them up with praise like "At least we have job!" & "For Mother Russia!" so it all turned out well in end.
Another fine example of the Russian Reversal.
Have gnu, will travel.
Lev Andropov: It's stuck, yes?
Watts: Back off! You don't know the components!
Lev Andropov: [annoyed] Components. American components, Russian Components, ALL MADE IN TAIWAN!
I was wondering when someone would get around to quality control on the subject line. I guess it might as well be me.
You made a heck of a lot of good points in a short post. Herr Hitler was democratically elected twice by the populace and his movement was financed by industrialists (capitalists) who were afraid the Reds were going to take over the country. Before they stepped in the help out the infant Nazi party Hitler was just some ex corporal who had served jail time. Those brown shirts cost money and someone had to foot the bill!
In return for keeping the communists at bay and advancing his cause, Hitler helped out the industrialists by building a lot of tanks, planes, and ships. Unemployment drops to near zero as people are either in the armed services or the war industries. Everyone wins! Well, with the exception of the people who are conquered or sent off to camps.