Slashdot Mirror
NASA Finds Critical Assembly Fault in Shuttle
mzs writes "During corrosion inspection on Discovery, technicians noticed that one of the gears in a rudder actuator had been installed backwards. This particular actuator was the top-most of four that control the air brakes on the tail. As luck turns out, if it had been the bottom-most actuator, loss of the shuttle and crew would have been nearly inevitable. Plans are in place to have four spares by the time Shuttle missions resume next year."
Could they not stamp "THIS SIDE UP" or whatever on the components?
Trolling is a art,
I thought they even checked Airplanes more thoroughly
Nothing to see here
...and I'm sure there will be lots of negative posts about NASA here...
It'd be nice to give some credit for the people that have put in layer upon layer upon layer of safeguards to check for exactly this sort of thing and the dilligent people that find this stuff. And caught it.
The awful thing is that this is going to be just another reason for Congress to loot the NASA money bag.
May we never see th
I'm quite surprised they're being quite so upfront about this. Kudos to them... On the other hand, I believe it to be a part of the healing process to convince the general public that they are, in fact taking the Columbia disaster extremely seriously, and want to show progress in the inspection and faliure-cathing procedures that obviously did not work for Columbia.
It was, however, just a matter of time before a Columbia-type disaster occured. The suttle program has a remarkable safety record, Challenger and Columbia no matter.
Didn't get the memo. I'm gonna go ahead and get you another copy of that, mmmkay?
Come on NASA, it's not rocket science! Oh wait...
I cannot believe that such a fundamentally problematic organization goes about its business mishap after mishap, without some high-level heads rolling every once in a while. Organizations get sloppy when they are not held accountable. To think that so many billions of taxes go toward what is supposedly one of our most high-tech endeavors, and they can't even install the parts correctly? Someone high-up should get fired for not forcing NASA to get serious.
---
http://thewired.blogs.com/teotwawki
The techno-mediated cultural conspiracy
...I'll be running Duke Nukem: Forever on Microsoft Longhorn before the next shuttle launches.
So which inspectors found the fault? Was it one of NASA's inspectors or one that was outsourced to India? Can somebody post a copy of the article please?
This is yet another reason that manned missions should be using simple reusable capsules instead of winged orbiters. There are no rudders to jam.
I'm paraphrasing here but it went something like this:
"When the most intelligent work on the most complex to build the the only prototype, inevitably the radio won't work."
The point is that when working on very complex designs and prototypes installing something incorrectly doesn't seem odd because your brain is unable to "see" the mistake for what it is. In a car, if you install the brakes incorrectly, the scale is such that you understand the mistake simply from your gut, visually. Like looking at a crumpled front fender and understanding that's not correct.
"This isn't a study in computer science, its a study in human behavior"
With all the advances in vehicle health monitoring, diagnostics, prognostics and the like it might be better for them to either build a new vehicle with this technology or retrofit the shuttles with it. Then they could see when the gears are cracked or acting up.
Evolution or ID?
NASA Finds Critical Assembly Fault in Shuttle
I know NASA is conservative with technology, but using assembly in this day and age is way backwards!
They should really do some double checking on this stuff. It's hard to imagine mistakes like this happen when dealing with something that holds the fate of a handfull of people's lives; not to mention all the millions of dollars put into these projects that would go down the drain. When dealing with people's lives and huge sums of money it's worth it to go over _everything_, and put in for better training so these thing don't happen again. They caught it this time, but if they don't take enough precaution, they might not be so lucky in the future.
Buckethead
And if you read the article, you realize that NASA installed defective actuators not once, but twice! The first being the one that was successfully flown 30 times, and the second in the spare actuators.
Given the complexity of a system like the shuttle, it is not surprising that out of 1000s of components there could be a mistake in one of them (and given some redundancy and robustness, it is not surprising that the shuttle could fly 30 times with one or more poorly installed components, though one would not normally want to bet on that...).
However, two errors out of 8 actuators checked implies some serious quality control issues.
-Marcus
Discovery flew safely 30 times with the defective actuator
When does a defect become a problem? I wonder if this was really a Critical problem because shouldn't some indication have already been seen by now?
I mean since they have fixed this problem will two other problems surface that are more critical and maybe they should have left it alone?
"whoops!"
Is anyone else reminded of the story of how Murphy's Law came into being (where something could be connected up the wrong way round and was)? I'm sure NASA has tightened up its procedures since Challenger/Columbia, but given that these things could be fitted either way it was an accident waiting to happen - thankfully it never did.
Isn't it about time they switched from assembly to C ?
NASA needs to start outsourcing to India, I hear they do great work for their pay.
I always thought there were 5 of everything to keep surfaces working even after a double failure. With only 4 actuators, if 2 fail, and start working against the other 2, the working pair can't overpower the non-working pair and the surface is useless. With 5 actuators, it takes a triple failure before the surface won't work.
cygnuhchur
The mistake dates back to the actuator's assembly at Hamilton Sundstrand in Rockford, Illinois, and is not easy to spot. The gear fits into the assembly both ways, but is slightly asymmetric so the teeth do not fit exactly if the gear is reversed.
Show me a man who can find a slightly asymmetric shape, and I'll show you a man who can find a slightly tritriangular number.
Or a slightly odd one ... hey wait, that's me. Except I am not a number, I am a free man!
I hear there's rumors on the Slashdots
In engineering, it's usually good practice to design somthing that only assembles one way. That way, whoever is assembling it (no matter how intelligent they are) can only install the component the way it was meant to be.
It's strange and somewhat disconcerting that this was not the case for this shuttle component, but I haven't seen the part in question.
ASANine
This whole thing is despicable.
The same basic thing happened with the F-111 program in the 60s. The drawing had a piece that was installed upside down, but the technician installing them said, this ALWAYS goes right side up and installed it that way. A couple crashes and the grounding of the whole F-111 fleet later, and the trouble was found. I don't know what happened to the installer, but I can't imagine it was any good. Check twice, install once :)
NASA Finds Hidden Shuttle Danger Same story, different article, in case the posted one gets /.'ed.
In this case nobody died and several lessons were learned, including something about fault-tolerance in actuators. I think two of the most valuable space flights from this point of view were Apollo 13 and the Mir mission that caught fire.
Things will go wrong. Learning how to cope when the evil wind blows is critical. In this case, we now know that the thing can be flown with one actuator in upside down. If the bottom one malfs, swap it out in orbit with the top one, and you still might get home. People are going to get killed doing this. People got killed learning to sail the Mediterranean. It's still worth doing.
Some mornings it's hardly worth chewing through the restraints to get out of bed.
I can imagine the guy that noticed this first. Probably went something like: looks at actuators. looks at diagram of how they're to be installed. looks at diagram again. looks at actuators. turns diagram around; notices that the legend is now upside down, so concludes that can't be it. checks other pages of diagram to see if this page is unusual--different view, maybe. finds that it isn't. checks back for errata. finds none.
Looks around. "Hey Bob, what do you make of this?" Thinks about all the work that day that isn't going to get done, because now management and, if he's lucky, congressional inspectors are going to crawl up his ass. At least he knows that he didn't *install* the things.
--
$tar -xvf
The gears were in an actuator that is, itself, a failsafe. It's apparently not used except in an emergency. That's it didn't fail in use; it was never used! The gears apparently are made to fit in either a right side or left side actuator but need to be installed with the proper orientation. Makes sense to use the same gear for both sides only flip it over. If its orientation is critical though, you'd better have some really good assembly instructions. Maybe like those that come with that high quality Chinese press board furniture!
...this sort of thing no longer happens (for NASA or Microsoft) is to put punitivie punchiments on the engineer/developer's heads if these things happen agaion.
If the USA is to become the empire it plans to be, we need more draconian measures to keep the incompitentes out of the way. Something like this would suffice:
From: Microsoft Corporate
Subject: Trusted Computing Initiative 2
In an effort to better secure computing for our customers, we are implementing new measures in our code revision system. The biometric login tubules installed at your workstations will allow you to log in and code the next gratest version of Windows OS. However, from now on, all code you write will be linked to your DNA. If, at some future time, a hole is found and your code is responsible, you will be summarily executed and replaced with the next coder in line. While this may sound a little harsh, we've found that this is the only way in which we can write secure code and still remain a proprietary OS with none of that dirty, smelly, nasty communist "free software" stuff going on. So go out and fix those bugs... your life is DEPENDING on it.
Sincerely,
Steve "The Fat Man" Ballme(r)
maybe one part of NASA was using radians, and another degrees.
Why does the modern Soyuz have a better safety record than the Shuttle? Why did our old ballistic missiles have a better safety record than the Shuttle? Even the enormous Saturn V rockets never had an accident in flight.[1]
Why does the Shuttle have such a terrible safety record relative to other rockets that attain orbit?
I'll tell you why: because it was over-ambitious. Congress was sold on the idea of a re-usable (read: cheap) launch vehicle that can do cool stuff like repair satellites. The truth of the matter is that if we had stuck with traditional launch vehicles (fire-once rockets), the money we saved over the long run would have allowed us to just replace failing satellites rather than repair them. (How many satellites have we repaired anyway?) We could even have built the space station for less. (Look at how we launched Skylab. Surely we could have repeated that a few times to get as large a space station as we wanted.) The legacy of the Shuttle is that of an overpriced, underperforming safety hazard.
All manned spaceflight is dangerous. The Shuttle is just more dangerous that most.
[1] The Apollo capsule had two serious accidents, one on the ground and one on the way to the moon.
Gates' Law: Every 18 months, the speed of software halves.
I work in New England, contracting for a jet engine manufacturer (and you can get it in two if you know the aero industry). Things like this happen frequently in manufacturing, especially with development hardware, before the kinks have been worked out of the assembly process and parts are ready to go to production. Assembly mistakes range from things that are easy to do but also easy to fix, like cut or cracked O-rings and tool knicks on non-critical parts, to things that are real screw-ups and result in major headaches, like parts left out entirely or vital parts being installed incorrectly and badly damaged because of it. You could consider the entire shuttle program to still be development-phase engineering, since only a few shuttles were ever built.
An example: a while back, we had a test engine spewing fuel all over the test cell for no readily apparent reason, prompting a panic that an entire compartment of the engine would have to be redesigned from scratch--until one of the test engineers found a fuel line seal that had not been reinstalled in the engine after the last teardown and reassembly. How do you miss something like this when there's a careful set of instructions to follow for every step of the assembly? I don't know, but I do know that humans are fallible, so we are constantly dealing with a stream of lost, damaged, and defective parts. Anyway, they put the seal back in, and the engine worked fine. (I have an NDA, so this is not what actually happened, but it is analogous.)
When I was in school, the more I learned about the environment the shuttle operates in, the more I was impressed by the fact that it worked at all, and now that I'm learning more about manufacturing engineering (not what I studied for; stupid job market), I'm surprised that the shuttles have as few problems as they do.
-Carolyn
Like Daddy always said: if you can't dazzle 'em with brilliance, baffle 'em with bullshit.
This seems to happen a lot with factory workers. As I recall from the fog of memory, Chuck Yeager talked about this one old guy working on the assembly line building F-86's. His job was to rivet on an actuator attachment point for the aileron's or some such. Then one day they redesign the whole actuator mechanism and the attachment point has to be flipped over and installed the other way. Well, this old rivetter guy refused to believe it. He thought the change order was a mistake, so he ignored it and kept putting 'em in the old way. Killed a few F-86 pilots before they figured out the problem.
Conclusion: the Empire squashes the Federation like a bug. Accept it.
On a side-note, the reason Nasa is stuck in the proverbial hard-place between multi-billion dollar budgets and missions that nobody cares about is that we've all started over-valuing human-life. It's ridiculous that space exploration all but stopped because of the 2 shuttle disasters. Certainly, the loss of those crews was tragic, but the best way to honor those crews is to relentlessly pursue the dream that they died for, not hamstring ourselves being overly cautious.
Call me old-fashioned, but I still believe there are things more important than one or a dozen human lives. IMO, exploring the universe is one of them.
The meek shall inherit the earth, in 3 by 6 plots. - Lazerus Long
The shuttle is the most complex system ever engineered by people... by orders of magnitude.
It's not suprising that there are flaws in the system - disasters lying dormant until the moment when they cause the destruction of the entire system.
This is one of the biggest arguements for a Vertical Takoff / Vertical Landing vehicle - it simplifies the system because it eliminates specialized components for landing.
Here's the mantra: fault tolerant systems. Things will fail. Can your space shuttle deal with those failures gracefully?
1. 2.
The article said that the teeth don't engage properly.
On high-load gears, the teeth are sometimes designed so that the faces which mesh are perpendicular to the force they apply. This keeps the gears from pushing each other away when they are loaded, and makes the gears engage more positively. But as a consequence, the teeth cannot be perfectly symmetrical.
If one of the gears is installed upside down, then the teeth would be loaded on a smaller surface area than designed (since their faces are now not parallel) which could cause them to deform or fracture. Also, since they now do push each other away, they could simply slip.
Obviously, if they fracture and leave a piece of metal free inside the gearbox, that could lead to a lockup.
MM
--
By including this sig, the copyright holders of this work or collection unreservedly place it in the public domain.
Great idea, but the workers stamped the wrong side on some versions, and the part cannot be remade in time.
This is a real problem in industry, you can put any sign on something, but then you gotta make sure the signs are right too. Indeed the wrong sign leads some workers to put it in backwards, even knowing the right way, while others will get in the habbit of putting it in with the lettering wrong, and not correct themselves when the next version is ships with the right parts.
That came from Chuck Yeager's autobiography. There was a line worker at the factory that was installing bolts in the wing that were supposed to be place head down/nut up. The worker was installing them head up/nut down contrary to the instructions because "that is the way bolts are supposed to be installed". IIRC he killed 6 people because of his ignorance.
Gee, where were you with your 20/20 hindsight vision when the Shuttle was being developed? It's very easy to make these kinds of statements now, 25 years after the fact. At the time though, there was a lot of pressure to make space flight cheap and prevalent and looking at that kind of volume a reuseable craft makes much more sense. Please don't treat the early shuttle designers as though they were money grubbing morons. There were many good reasons to do what they did, and there are many things, like Hubble, that would have been much harder to launch and maintain without the shuttle.
THIS SPACE FOR RENT
...I remember as much from "Engine parts" that depending on load, you'd like the teeth of the gear to NOT be symmetric. This would give you better interconnection in one direction (the "right" way) and worse in the other (the "wrong" way).
Ever see a winch? The teeth on the gear there is an extreme of that sort - only designed to pull load in. So it's not done to be mean - it's probably done as to fit the spec.
Kjella
Live today, because you never know what tomorrow brings
I love watching the comedy of shuttle sustainment. Remember how it was sold to us years ago: As a quick turn bird. Instead, after each flight they even change out whole components ( certain back up systems ) that sometimes never get used on a mission. Inspect the item, if it is good to go, don't mess with it. Sign it off and press on. Lots of things on the shuttle aren't complex ( some are ). But don't fix items that are not broken. Just thought I would mention this because that is the culture there. Now in the case of the vertical stab, I am sure that thing takes a beating every mission and is worth refirbing. Be good to take some of the manpower and sustainment resources from stuff that isn't broken and doesn't need a refirb after every flight and apply it to the items that do. Might even reduce the total ( large ) number of flow days for an airframe between missions. Airframe sustainment issues over time, are very fun and not all that hard to grasp. I am sure there are other fun fubars that we haven't heard about. Hey, somebody caught it at least.
ANY system where the right side up/down of a single fucking nut put on by one single guy that does not have any kind of independent quality control/inspection/etc process, any system at all that allows a bolt put on backwards by one single guy to kill 6 people...
It's a systemic fault.