Slashdot Mirror
NASA Finds Critical Assembly Fault in Shuttle
mzs writes "During corrosion inspection on Discovery, technicians noticed that one of the gears in a rudder actuator had been installed backwards. This particular actuator was the top-most of four that control the air brakes on the tail. As luck turns out, if it had been the bottom-most actuator, loss of the shuttle and crew would have been nearly inevitable. Plans are in place to have four spares by the time Shuttle missions resume next year."
I thought they even checked Airplanes more thoroughly
Nothing to see here
...and I'm sure there will be lots of negative posts about NASA here...
It'd be nice to give some credit for the people that have put in layer upon layer upon layer of safeguards to check for exactly this sort of thing and the dilligent people that find this stuff. And caught it.
The awful thing is that this is going to be just another reason for Congress to loot the NASA money bag.
May we never see th
I'm quite surprised they're being quite so upfront about this. Kudos to them... On the other hand, I believe it to be a part of the healing process to convince the general public that they are, in fact taking the Columbia disaster extremely seriously, and want to show progress in the inspection and faliure-cathing procedures that obviously did not work for Columbia.
It was, however, just a matter of time before a Columbia-type disaster occured. The suttle program has a remarkable safety record, Challenger and Columbia no matter.
I cannot believe that such a fundamentally problematic organization goes about its business mishap after mishap, without some high-level heads rolling every once in a while. Organizations get sloppy when they are not held accountable. To think that so many billions of taxes go toward what is supposedly one of our most high-tech endeavors, and they can't even install the parts correctly? Someone high-up should get fired for not forcing NASA to get serious.
---
http://thewired.blogs.com/teotwawki
The techno-mediated cultural conspiracy
I'm paraphrasing here but it went something like this:
"When the most intelligent work on the most complex to build the the only prototype, inevitably the radio won't work."
The point is that when working on very complex designs and prototypes installing something incorrectly doesn't seem odd because your brain is unable to "see" the mistake for what it is. In a car, if you install the brakes incorrectly, the scale is such that you understand the mistake simply from your gut, visually. Like looking at a crumpled front fender and understanding that's not correct.
"This isn't a study in computer science, its a study in human behavior"
With all the advances in vehicle health monitoring, diagnostics, prognostics and the like it might be better for them to either build a new vehicle with this technology or retrofit the shuttles with it. Then they could see when the gears are cracked or acting up.
Evolution or ID?
They should really do some double checking on this stuff. It's hard to imagine mistakes like this happen when dealing with something that holds the fate of a handfull of people's lives; not to mention all the millions of dollars put into these projects that would go down the drain. When dealing with people's lives and huge sums of money it's worth it to go over _everything_, and put in for better training so these thing don't happen again. They caught it this time, but if they don't take enough precaution, they might not be so lucky in the future.
Buckethead
And if you read the article, you realize that NASA installed defective actuators not once, but twice! The first being the one that was successfully flown 30 times, and the second in the spare actuators.
Given the complexity of a system like the shuttle, it is not surprising that out of 1000s of components there could be a mistake in one of them (and given some redundancy and robustness, it is not surprising that the shuttle could fly 30 times with one or more poorly installed components, though one would not normally want to bet on that...).
However, two errors out of 8 actuators checked implies some serious quality control issues.
-Marcus
I always thought there were 5 of everything to keep surfaces working even after a double failure. With only 4 actuators, if 2 fail, and start working against the other 2, the working pair can't overpower the non-working pair and the surface is useless. With 5 actuators, it takes a triple failure before the surface won't work.
cygnuhchur
The mistake dates back to the actuator's assembly at Hamilton Sundstrand in Rockford, Illinois, and is not easy to spot. The gear fits into the assembly both ways, but is slightly asymmetric so the teeth do not fit exactly if the gear is reversed.
Show me a man who can find a slightly asymmetric shape, and I'll show you a man who can find a slightly tritriangular number.
Or a slightly odd one ... hey wait, that's me. Except I am not a number, I am a free man!
I hear there's rumors on the Slashdots
Quoth John Carmack
... There's this mytholigization of aerospace that it's the hardest thing you can do. That's just not true. In terms of actual difficulty, it's not that hard. Aerospace is plumbing with the volume turned up."
"I'm kind of at the top of my field [in gaming]," he said. "When I started reading about aerospace, I realized there was an incredible level of things to learn.
The same basic thing happened with the F-111 program in the 60s. The drawing had a piece that was installed upside down, but the technician installing them said, this ALWAYS goes right side up and installed it that way. A couple crashes and the grounding of the whole F-111 fleet later, and the trouble was found. I don't know what happened to the installer, but I can't imagine it was any good. Check twice, install once :)
In this case nobody died and several lessons were learned, including something about fault-tolerance in actuators. I think two of the most valuable space flights from this point of view were Apollo 13 and the Mir mission that caught fire.
Things will go wrong. Learning how to cope when the evil wind blows is critical. In this case, we now know that the thing can be flown with one actuator in upside down. If the bottom one malfs, swap it out in orbit with the top one, and you still might get home. People are going to get killed doing this. People got killed learning to sail the Mediterranean. It's still worth doing.
Some mornings it's hardly worth chewing through the restraints to get out of bed.
I can imagine the guy that noticed this first. Probably went something like: looks at actuators. looks at diagram of how they're to be installed. looks at diagram again. looks at actuators. turns diagram around; notices that the legend is now upside down, so concludes that can't be it. checks other pages of diagram to see if this page is unusual--different view, maybe. finds that it isn't. checks back for errata. finds none.
Looks around. "Hey Bob, what do you make of this?" Thinks about all the work that day that isn't going to get done, because now management and, if he's lucky, congressional inspectors are going to crawl up his ass. At least he knows that he didn't *install* the things.
--
$tar -xvf
The gears were in an actuator that is, itself, a failsafe. It's apparently not used except in an emergency. That's it didn't fail in use; it was never used! The gears apparently are made to fit in either a right side or left side actuator but need to be installed with the proper orientation. Makes sense to use the same gear for both sides only flip it over. If its orientation is critical though, you'd better have some really good assembly instructions. Maybe like those that come with that high quality Chinese press board furniture!
This is the original definition of Murphy's Law: If it is possible to do something in more than one way, and one of those ways leads to catastrophe, someone will inevitably do it the wrong way.
In fact the original inspiration for Murphy's Law was a G-force meter that was installed backwards, thereby taking meaningless readings. (It probably didn't go below zero.)
Another example is the 1969 gearbox fire on the Canadian navy ship, HMCS Kootenay. A gearbox bearing was installed backwards, which restricted its flow of lubrication oil (on a naval vessel, the gearbox is the size of a car and absorbs tremendous loads). Apparently it did say which way to install it, but the installation was made in a foreign shipyard where the workers could not read English. The poorly lubricated bearing overheated and caused an explosion during a full-power trial; nine sailors were killed and dozens injured.
The moral of all these stories is: if it's important which way something is installed, make it asymmetric so that it's physically impossible to install it the wrong way. Labels are not enough.
Toronto-area transit rider? Rate your ride.
Dunno about the mortars, but current claymore mines are stamped as such. On the front curved side, its imprinted:
FRONT TOWARDS ENEMY.
Here is a pic
"The natural progress of things is for liberty to yield and government to gain ground." - Thomas Jefferson
I work in New England, contracting for a jet engine manufacturer (and you can get it in two if you know the aero industry). Things like this happen frequently in manufacturing, especially with development hardware, before the kinks have been worked out of the assembly process and parts are ready to go to production. Assembly mistakes range from things that are easy to do but also easy to fix, like cut or cracked O-rings and tool knicks on non-critical parts, to things that are real screw-ups and result in major headaches, like parts left out entirely or vital parts being installed incorrectly and badly damaged because of it. You could consider the entire shuttle program to still be development-phase engineering, since only a few shuttles were ever built.
An example: a while back, we had a test engine spewing fuel all over the test cell for no readily apparent reason, prompting a panic that an entire compartment of the engine would have to be redesigned from scratch--until one of the test engineers found a fuel line seal that had not been reinstalled in the engine after the last teardown and reassembly. How do you miss something like this when there's a careful set of instructions to follow for every step of the assembly? I don't know, but I do know that humans are fallible, so we are constantly dealing with a stream of lost, damaged, and defective parts. Anyway, they put the seal back in, and the engine worked fine. (I have an NDA, so this is not what actually happened, but it is analogous.)
When I was in school, the more I learned about the environment the shuttle operates in, the more I was impressed by the fact that it worked at all, and now that I'm learning more about manufacturing engineering (not what I studied for; stupid job market), I'm surprised that the shuttles have as few problems as they do.
-Carolyn
Like Daddy always said: if you can't dazzle 'em with brilliance, baffle 'em with bullshit.
This seems to happen a lot with factory workers. As I recall from the fog of memory, Chuck Yeager talked about this one old guy working on the assembly line building F-86's. His job was to rivet on an actuator attachment point for the aileron's or some such. Then one day they redesign the whole actuator mechanism and the attachment point has to be flipped over and installed the other way. Well, this old rivetter guy refused to believe it. He thought the change order was a mistake, so he ignored it and kept putting 'em in the old way. Killed a few F-86 pilots before they figured out the problem.
Conclusion: the Empire squashes the Federation like a bug. Accept it.
On a side-note, the reason Nasa is stuck in the proverbial hard-place between multi-billion dollar budgets and missions that nobody cares about is that we've all started over-valuing human-life. It's ridiculous that space exploration all but stopped because of the 2 shuttle disasters. Certainly, the loss of those crews was tragic, but the best way to honor those crews is to relentlessly pursue the dream that they died for, not hamstring ourselves being overly cautious.
Call me old-fashioned, but I still believe there are things more important than one or a dozen human lives. IMO, exploring the universe is one of them.
The meek shall inherit the earth, in 3 by 6 plots. - Lazerus Long
I'm not a mechanical engineer either, but I know that metal parts are usually painted to prevent corrosion. In fact, that's why these were removed--to check for corrosion. I guess it's lucky they didn't paint them or they might never have noticed it was in backwards....
So now they resort to cannibalization to fix the shuttle.
Makes you feel REAL secure about it.
It implies that they cannot manufacture the necessary spare parts.
He blames it on the ten thousand different manufacturers you deal with in IT, ranging from motherboard suppliers, to RAM makers, to CPU makers, hard drives, UPSs, and of course, software. The pieces work, it's getting them to work together that's a bitch. With a satellite, you have maybe 20 or 30 people who, in combination, know everything about it and who can coordinate with each other.
Think about that. Keeping a mid-size server farm up and running smoothly (all the while undergoing constant upgrades, new feature additions, etc.) is more difficult than designing and launching a satellite. Straight from the horse's mouth.
The article said that the teeth don't engage properly.
On high-load gears, the teeth are sometimes designed so that the faces which mesh are perpendicular to the force they apply. This keeps the gears from pushing each other away when they are loaded, and makes the gears engage more positively. But as a consequence, the teeth cannot be perfectly symmetrical.
If one of the gears is installed upside down, then the teeth would be loaded on a smaller surface area than designed (since their faces are now not parallel) which could cause them to deform or fracture. Also, since they now do push each other away, they could simply slip.
Obviously, if they fracture and leave a piece of metal free inside the gearbox, that could lead to a lockup.
MM
--
By including this sig, the copyright holders of this work or collection unreservedly place it in the public domain.
Ah, but this has been tried elsewhere. Look up the design history of the swing-wing pivots on the B-1 bomber. The pivots are two large hollow metal cylinders, slightly asymmetrical. One side is clearly marked "THIS SIDE UP."
However, the manufacturer stamped the wrong side of the pivot, and it was dutifully installed upside-down. Murphy wins again no matter what you do.
In the end they will lay their freedom at our feet and say to us, Make us your slaves, but feed us. - Fyodor Dostoyevsky
I love watching the comedy of shuttle sustainment. Remember how it was sold to us years ago: As a quick turn bird. Instead, after each flight they even change out whole components ( certain back up systems ) that sometimes never get used on a mission. Inspect the item, if it is good to go, don't mess with it. Sign it off and press on. Lots of things on the shuttle aren't complex ( some are ). But don't fix items that are not broken. Just thought I would mention this because that is the culture there. Now in the case of the vertical stab, I am sure that thing takes a beating every mission and is worth refirbing. Be good to take some of the manpower and sustainment resources from stuff that isn't broken and doesn't need a refirb after every flight and apply it to the items that do. Might even reduce the total ( large ) number of flow days for an airframe between missions. Airframe sustainment issues over time, are very fun and not all that hard to grasp. I am sure there are other fun fubars that we haven't heard about. Hey, somebody caught it at least.
Not even close. Try a nuc powered aircraft carrier. With more plumbing per square yard than anything, counting fuel, lube oil, hydraulics, steam, water, waste, firefighting......... Largest and most comples by far.
Professional Politicians are not the solution, they ARE the problem.