Slashdot Mirror
Sequence of Events During Columbia Mission
applemasker writes "Today's NYT is reporting that NASA managers actively resisted requests from vehicle engineers for on-orbit imagery. This should answer Administrator O'Keefe's question of why no engineers 'spoke up' during the flight. Seems they did; managers just ignored them."
"... managers just ignored them."
The story of an engineer's life.
steve
Oh, you're not stuck, you're just unable to let go of the onion rings.
Engineers make recommendations. Managers disregard them. Things like impressing VPs, etc are way more important to get ahead in an organisation unfortunately.
...is that all of the managers on the mission, including Ms. Ham, have apparently been reassigned or they've retired. The behavior quoted in the article (assuming it's accurate), is inexcusable.
Tim
Can those managers be charged with manslaughter now?
This sounds like what happens with any career where the management doesn't know as much as the subordinates. As such this should send the message out that when someone tells you that something is a bad idea then you might want to consider why they say its a bad idea. After all how many of us have had our boss(es) tell us to do something that is either technically not possible (for any reason), or is dangerous?
Now what's next? Managers should be expected to listen to engineers???
Here's a BBC story on the same subject.
For those like me who do not wish to register with NYT
This is just another example of one of my favorite statements:
"A bad technical decision is never a good business decision!"
Regardless of the circumstances, a bad technical decision is never good from a business perspective. It never cost less in the long run. A mediocre decision may be a good one because of cost, but a bad one will fail, cost you more, and failure is never good for business. Unfortunately too often managers don't understand this until it's too late.
The panel findings that NASA was starved for funds by congress and the White House. One congress man actually said that "The problems at NASA would still exists even if we gave them a blank check."
No they wouldn't.
The managers are told that they have to fly x number of missions on x number of dollars. If they fly less they get even less money.
Don't blame the managers blame congress and the last couple of Admins. Yes Billy Boy during a time of budget surplus never gave NASA a buget increase.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
Do you launch another shuttle mission, have both dock at the space station? Do you set up a moon base? Do you develop a new low-orbit rescue vehicle? Does everyone moonwalk from one shuttle to another? Do we redesign the shuttle to have a safty escape module that can blast loose of the mother ship and safely return to earth?
I bootleg Fizzy Lifting Drinks.
Welcome to The New York Times on the Web!
For full access to our site, please complete this simple registration form.
As a member, you'll enjoy:
In-depth coverage and analysis of news events from The New York Times FREE
Up-to-the-minute breaking news and developing stories FREE
Exclusive Web-only features, classifieds, tools, multimedia and much, much more FREE
When I was taking the required Technical Communication course in college to finish my engineering degree, a major theme of the class was incidents such as Three Mile Island and the Challenger disaster. The professor said that while the public perception was that management had f***ed up, the engineers had to bear some responsibility because they were unable to adequately communicate the necessary conclusions in a manner that decision makers could understand. And we would look at copies of the memos, and think that, yeah, if the engineers had written more effectively, things may have been different.
In some ways, even though I don't enjoy writing specs and design documents for software (I don't work on mission-critical or life-critical systems), I try to write well, because I figure, "I'm an engineer, and I have a responsibility to do my job as a professional."
And then I read this article, and I think that maybe, after all, it doesn't matter what a competent, professional engineer says or does. I'm just saddened that NASA, an institution I loved growing up, did not change at all after Challenger. I wish I knew the answer.
Actually the military / CIA mislead the manager of the shuttle program about the capabilities of the satellites because he didn't have the required security clearance. He therefore determined that the images wouldn't be of sufficient quality to find a possible problem.
This was in one of the reports from the investigation board.
How about giving them an opportunity to say goodbye to their friends and family?
I'd say that's worth it.
If NASA managers listened to every issue brought forward by each of their thousands of engineers, spaceships would never leave the earth.
It's in each of these engineers best interest to list every problem that could possible occur in the systems they design and maintain. That way if the problem happens in one of their systems, they can cover their ass with paperwork. Just because they issued a low-level memorandum doesn't mean these engineers actually had any level of confidence that the problem would occur. It just meant they were covering their ass.
NASA has an escalation process, if these engineers *really* felt there was a problem, all they had to do was push that button. But to push the button is to put your clout on the line. Push it too often by mistake and you will rightly be taken out of the process. No company or organization can afford an employee that continually cries wolf.
So if anyone is to blame for this, it's not the managers. It's the engineers that wrote memo's about it to cover their ass but didn't think the problem was important enough to push the escalation button.
The managers are so inundated with engineers thinking up possible error scenarios they can't possible take them all seriously. Of course, when a shuttle goes down, those same engineers drag out the paper trail covering their butt and program managers are left to swing.
Congress should be ashamed of this inquiry and so should most of America. Space travel is dangerous. Live with the danger or get out of the business.
Absoltuly false. I am in a management profession, and I can tell you from experience that people that do this are only successful to a point.
What you said is the recipe to make it into middle managment (Director or VP level), but you will never get beyond that. Company Sr. Executives and Officers are expected to be frank and honest. Those that aren't generally don't fare well (yes, you can point out exceptions, but as a general rule, liars don't make it to the top).
Unfortunatly, honest senior managers often have kiss-ass middle managers working for them. Those middle-managers lie, cheat, steal, etc, and senior management is left holding the bag for their mistakes (which is the job of management, to take the fall when your subordinates screw up, in case any managers reading this have forgotten... ignorance is not\ excuse).
I made it into senior management very quickly in my career by having a policy of never breaking the law, never lying to my boss and never sucking up to anyone. Say what you want, but those simple ethics, combined with extremely hard work, are what put me on the fast track. Managers who will screw people to get ahead will find their careers never make it to where they could (but probably does go past where they should).
Sarcasm and hyperbole are the final refuges for weak minds
This results in a culture where we promote a weakness (no communication skills) as a virtue (disdain for politics). The only way to change this, I think, is to emphasize writing and communications coursework as well as courses where you learn how not to kill people by leaving a bolt off the diagram.
Except that without knowing what the extent of the damage is in the first place, it's impossible to determine if it can be repaired in the first place. So perhaps there might have been a plausible repair scenario (or at least the opportunity to do something that didn't involve the death of a shuttle crew), but since no investigation was done while the opportunity was avaliable. NASA might be a godawful bureaucracy, but if you strip away the bureaucrats, you're left with people who have something of a clue and could have worked out something, instead of pretending that the problem didn't exist in the first place.
Dogma: Dead (mostly because your Karma ran it over)
It's the same thing in the computer/IT sector. More and more the management has no technical skills, just business skills. But these people are the ones who decide what technology is best. Why the NASA management wouldn't point a telescope at the Shuttle when engineers felt there was need for more information is beyond me. Most likely it was purely a financial decision.
r t people running around the industry now. But that has nothing to do with the Shuttle and NASA. Those engineers were/are capable of the tasks at hand.
15 years ago, it was very common for technical people to fill management positions up through middle management with the Chief Engineer over seeing all the technical departments and reporting directly to the top level management. Today, we're luck to get technical expertise beyond the department/group managment level.
This isn't a NASA-only problem. It's an industry wide problem. For example, the CSX RailRoad had it's signaling system go down because the computers running all those signals runs Microsoft Windows and got a virus. Who but a non-technical managager would insist Windows be used in a mission critical task like this? This might not be a good example because I have no proof it was a management decision while it very well be a technical moron made the choice and dumb PHB's followed the advice. The choice should not have been followed if a technically savy management existed.
There's also been a dumbing down of the technical sector with all these I-can-click-an-icon-therefore-I'm-a-computer-expe
Does anybody else think that management making technical decisions no longer make them with much regard to input from the engineers anymore?
LoB
"Anyone who stands out in the middle of a road looks like roadkill to me." --Linus
What's most interesting to me about this story is that both the engineers and the managers were making the best decision based on their perspectives. The engineer's perspective is based on hard facts, information, and analysis. The manager's perspective is based on people issues: money, resource management, risk management, project deadlines, etc.
It's easy in retrospect to criticize managers who didn't want to be a "Chicken Little" or who, upon getting feedback from upper management, called it a "dead issue". But if they had gone ahead with the imaging, and the photos showed no damage and the shuttle had landed safely with no (or insignificant) damage to the wing, their reputation would have suffered. They would have been faulted for allocating valuable resources on something that turned out not to be an issue.
Part of a manager's job includes risk management and resource allocation. This means properly assessing the likelihood and impact of a risk. In this case, I would suggest that management considered the 'cost' of pursuing further investigation to be higher than the 'likelihood * impact' factor of doing nothing. They have probably made the same decision many times before, successfully, which would encourage them to make the same decision again. Only this time, they were wrong -- the statistics caught up with them.
-Thomas
When the MsBlast worm hit our place in August and I saw the Slashdot story, I saw a spike in our call volume about two minutes before. I immediately notified my manager and told her that something needed to be done. She said, "huh, what's slashdot?" called her manager and said an employee got a message off some unauthorized site. Then she promptly did nothing.
We are still taking calls about that virus, and the bass ackwards crap they did to remedy the fallout. Managers are paid to make a team go in a direction and be productive. Not to ignore the people they "manage". Part of being productive is knowing that you listen to your team.
I can kind of sympathize with dumb managers though. If everyone who thought there was a major issue came to them and bitched their ears off, they'd never get anything done. Adding another layer between the management and team seems asinine too, because inevitably there just become too many layers to communicate through. As evidenced in the article, where Mr. Rocha ignored protocol and wrote directly to the head honcho of NASA (god forbid!). I think it goes to reinforce the fact that business managers and people who go to business school to become managers are worthless. Moving up through the ranks and cutting your teeth is the only way to find a good manager who will consistently know when a team member is talking out there ass or should worry when confronted.
Oh, well, I guess one day I'll have seniority, over somebody, somewhere, somehow.
Welcome your new Slashdot overlord non-sig.
So an engineer saw a problem and was concerned. My question is how often does this happen. If after every launch there are 100 engineers who noticed a potential problem, then I'd have ignored this too (along with the 99 other potential problems that didn't kill columbia) If enginneers almost never see a potential problem then this should have been taken seriously.
Others have pointed out that there is an esclation process for problems belived to be serious, and that wasn't followed. In hind site it should have been, but they didn't have hind site to work with then, so we have to be realistic i our expectations.
Part of the problem is the damage caused to NASA by years of budget cuts. I saw this first hand. Due to a lack of funds, NASA adopted an attitude that sustaining engineering and operations costs could be substantially reduced by avoiding change whenever possible. Just keep the current system running with as little maintenance as possible. If nothing changes, you can get rid of most of the people who used to design, test, document and maintain the systems. If there is a problem with a system, you don't find the root cause and fix it, you develop a work-around. If new technology offers a better way to do something, you ignore it because the old system is "good enough" and you no longer have the money, infrastructure and people needed for major design changes and new systems development. The organization gets reduced to a caretaker for the engineering accomplishments of previous generations. It has just enough money and people to maintain the status quo.
Mea navis aericumbens anguillis abundat
I read the entire Columbia report, and this article. Although I don't think we should always look for a person to blame after an accident, this was such a case of gross mismanagement that I really hope both Ham and Schomberg get at least a few months in "Club Fed" for their actions. Ham had future launch dates taking priority over her current mission. She quashed three requests for imaging personally, primarily because it would be the admittance of a problem that would throw the next mission off schedule. Schomberg on the other hand was just a poor engineer. He spouted off all week that he was the "EXPERT". Without doing a single calculation or having a shred of evidence, he just knew the Shuttle was safe regardless of what others said because he was the "expert". Sounds more like a petulant child to me.
"Curiosity killed the cat, but for a while I was a suspect."- Steven Wright
Columbia was mostly a science experiment mission. I heard a talk a month ago from the Principal Investigators of two experiments. Because these both had cameras and telemetry, they each returned 90% of their results. They were hoping to retrieve the apparatus for final analysis, but the pieces recovered after the accident weren't too useful. However, one of the experiment had got 5% additional results when a disk platter was forensically read after the accident.
Both investigators said the astronauts were crucial to the success of their experiments. Although they were supposed to be mostly automatic, Murphy's law intervened, and the astronauts had to help. One astronaut even devoted several hours of her recreation time to fixing a busted valve (The ground crew had stayed up 96 hours straight working on a solution). All of the ground material was impounded for two months after the accident to rule our experimental causes of the accident.
One result is of immediate use to NASA. It was a study of extinguishing fires with a new kind of water mist that could only be studied in microgravity. Since the prediction was successful, this means that water-based extinguishers could replace chemical extinguishers in space and on earth in more situations.
Overall 60% of the results on the entire missionwere successfully returned. Slightly more may be retrieved through forensics. I was surprised to hear this high a success.
It was not decided yet whether there would be a collective publication of their successful results as a memorial to the mission. They will of course publish in their respective journals.
Something I've been thinking about for some time...
It's hard to believe that the NASA managers ALL were indifferent to or ignorant of the potential damage to the shuttle. If you're an engineer, you can run through the numbers in your head in about 5 seconds flat: mass x velocity x surface area= pressure per square inch.
If you know anything about the shuttle, you know that the tiles are fragile and subject to fracture on impact (in fact a major worry always has been what happens if the Shuttle hit a piece of space junk.)
And if you know anything about the shuttle project, you also know that the crew had limited ability to fix a lot of things that might go wrong after the shuttle lifts off the pad.
So what if you're a manager with the big view and the big leather chair and an engineer or several come to you with concerns about the impact on the wing?
And you do the math in your head and remember that there are no spare tiles on board and basically if the wing has been holed, the crew cannot be saved?
Choice 1: Raise the alarm, go through an agonizing several weeks of total public panic/crisis until the shuttle runs out of food, fuel and/or life support and watch the crew die in front of the world? or,
Choice 2: Put a lid on it and let the shuttle go through its mission, hoping that a miracle might happen and the damage is not serious enough to cause breakup on reentry?
So the question is, what do you do?
___________
In other words, what NASA management knew it had only two choices and chose #2? and if they did, was that the wrong choice?
[Insert pretentious and semi-clever sig here: ______ ]
The issue is not "did NASA engineers raise concerns" but did they raise concerns above the level that usually triggers a more serious review. I am sure that on every single shuttle mission there were engineers that raised concerns about every single glitch, out-of-tolerance reading, or unusual occurence, etc. This is a good thing. It is also a good thing that other engineers and managers make informed cost-benefit decisions to either pursue, study, or ignore any raised concerns.
Hindsight is 20-20. Nobody remembers all the prior events in which engineers raised concerns that were ignored and nothing happened. Don't forget this was not the first time that insulation had fallen off the external tank. As an engineer myself, I know I can come up with all manner of "potential concerns." As an older engineer, I know that many of those concerns can easily fail a cost-benefit analysis or prove to be groundless on further study.
Tuning the process of raising and dispatching concerns is very hard -- being overly cautious is as damaging as being overly risky. It is especially hard with the extremely low sample sizes and highly complex systems that NASA faces when managing the shuttle. Personally, I am surprised that the shuttle is as reliable as it is.
I hope that NASA can keep flying because it is the only way that humanity can get the experience needed for truly reliable space flight in the future.
Two wrongs don't make a right, but three lefts do.
Yes, I agree. It sickens me that it has been almost 40 years since people landed on the moon, and the human component of space exploration is barely out of the atmosphere, and only done by a poorly funded govt. organization. It sickens me to read about Software patents in Europe, the USPTO here, the way John Ashcroft wants to police america, and all the wars and conflict in the world that we have the resources to resolve, but don't.
It sickens me that people care more about their own self interest and their company or agency's PR, than advancing the human race. I really hope in the next election some of these issues come up, instead of being pushed behind the tired old debates about abortion and taxes.
The sending of this message pretty much inconveniences everyone involved.
Still, wouldn't it be worth a shot? I mean, you'd pretty much know you would burn up trying to land the shuttle with a hole in the wing.
They say it didn't have enough fuel, but that was as loaded. I don't presume to know much about the shuttle, but surely they could jettison some equipment to reduce the mass. Experiments don't look very important when you know the shuttle is going to disintegrate upon re-entry; they're already gone!
Does anyone know if one of the Soyuz capsules could dock with the shuttle? If not, could they use the EVA to transfer. There are two suits, so it'd take a while to transfer everyone, but I think it'd be possible. Don't they have one at the space station already? I don't think it holds enough people for the crew of seven, but that would allow more time (less people = less consumed) to get another Soyuz or space shuttle up there. Still, that would leave the space station crew without an escape method, but it'd be worth the risk I think. Much safer than landing a space shuttle that we already know is compromised.
But the space program died out even after Apollo 13. The American public didn't want to support scientific exploration of space, they were only interested in beating the Russians to the moon.
Read her side of the spy-satellite picture story and watch for spin.
Click.
I've just been reading "What do you care what other people think" by Richard Feynman, and it covers some of his life during the Challenger investigation. And it was the same then as it is now...The field techs and engineers saying "This is really dangerous" and the Suits in management saying "But it worked before, why is it not safe now?!?!". It is a sad story about our Western Civilization that communication between the top and bottom of companies is so bad it is non-existant. If people in Management went and read the Toffler's Future Shock, and the books that come after it, they would understand why it is so important esp. in today's ultra-fast communication age that the heirarchy between the top and bottom of companies be flattened.
...who happily works for a company where the management *are* engineers and still to engineering work, and thus will listen to their workers.
Of course, if it was just money, it might not be that important...but PEOPLE DIED because managment didn't listen...and every day PEOPLE DIE because management continues to be def to the information comming from below.
ttyl
Farrell
CAN-CON 2019 - Ottawa's only book oriented Science Fiction Convention! October 18-20, Sheraton Hotel, Ottawa, Canada h
a. Tell the crew the situation, and let the crew decide whether to take a chance.
b. Breaking every single shuttle launch safety rule, try to launch another shuttle with 1-2 crew, and a bunch of space suits, before Columbia ran out of consumables.
Neither choice would look good to me.
Luckily your question is answered by the CAIB report. First, an ad-hoc wing repair using a combination of water (frozen in space), titanium tools on board the shuttle, and miscellanous junk might have held in place long enough to allow the shuttle to reenter without being destroyed. Second, by working around the clock in shifts, the next shuttle launch could have been moved up in time to rescue the Columbia with about 5 days to spare, without skipping any safety checks.
The CAIB report rejected the possibility of tranferring to the ISS (too much delta-V for the fuel left on board), and flying a different reentry pattern that would take load off of the damaged wing (too dangerous). Of course those were just the first four suggestions for approaches that might have been tried had they known that there was something wrong; no doubt there would have been dozens of other ideas floated if the engineers had had the need to do something.
--
BitTorrent in C -- LibBT
http://www.sf.net/projects/libbt
I do blame the managers and I do blame congress. I blame NASA for failing to be truthful in it's own cost and safety reports. I blame Congress for not providing sufficient oversight and for forcing sub-par designs on NASA in order to appease pork barrel political hand-outs.
Also, I fail to see how you can blame "Billy boy" when he was busy fighting off impeachment and harrassment by a Republican congress when GBush I didn't do diddly for NASA either.
There is no trap so deadly as the trap you set for yourself
-Raymond Chandler, The Long Goodbye
Why is this so hard to understand? Engineers are failure oriented. We look for ways to break stuff, and then plan to mitigate its breakage. We always look at the worst case scenario. I am an engineer, and I know the words, "Yeah, it won't break" have never passed my lips unless accompanied with several volumes of caveats.
Face it folks, engineers are sky-is-falling-folks. We could stand to filter ourselves a little bit to gain some credibility.
"Yeah, the engineers say something bad is going to happen, but they say that every day. Shall we launch, then? Okay, good to go."
I mean, if you say every single day, the world is going to end, and then one day it actually does, did you, in fact, predict it?
Toddlers are the stormtroopers of the Lord of Entropy.
Just because most people treat risk analysis like some grade school math problem doesn't mean that there hasn't been lots of research on how to do proper risk analyses for complex systems. It isn't simple, but you can do a rigorous risk analysis based on uncertain information. Such an analysis would show which missing information is contributing the largest amount of uncertainty to the end result. In this case, the largest uncertainty was "WHERE DID THE FOAM HIT". Given that this most basic uncertainty was never resolved until much later, there was no way that a proper analysis could have said with any certainty as to the safety of the Columbia given the foam strike.
"We think the foam was this big, we think it didn't hit a critical tile and we think our computer program is too pessimistic so the shuttle is safe" is utter and complete BULLSHIT. It doesn't matter how many numbers you wrap around those words. A bullshit perspective is still bullshit. And no real engineering manager would have let the Lockheed engineers get away with presenting the crappy analysis report. I have another post from a Feburary shuttle story about this.
There is no trap so deadly as the trap you set for yourself
-Raymond Chandler, The Long Goodbye
The Board states in the report that they asked NASA what could have done if the seriousness of the damage had been confirmed. NASA responded by staging a real-time simulation, assuming the discovery had been made late in the mission. They worked two solutions in parallel. One had a repair-on-orbit solution using materials on board _Columbia_. It included jettisoning most of the cargo and using a reentry which put a lower heat load on the damaged area. The other had _Atlantis_ making a rendezvous with a crew of four and docking equipment. With no major countdown holds and the _Columbia_ crew taking it easy, this could have been done before the last of their carbon dioxide absorbing cannisters was used up. The second alternative was by far the preferred, since they couldn't be sure the repair would hold. But it would probably have been done anyway, in case _Atlantis_ was late. Following crew recovery, the empty _Columbia_ would have been put into a reentry into the ocean, or boosted to a higher orbit for later repair. Stickmaker
Seastead this.
Engineers in the real world try to make things work. The biggest problem with this is managers who share your beliefs who believe that problems can be wished away by managerial fiat.
The escalation you whine about was blocked by the action of a bureaucrat at the wrong place and the wrong time, and people died.
This isn't an engineering problem, it's a business process problem and in general, the solution is finding management like you and terminating it and putting procedures in place which will make future managers of the type you support disappear. This is just as important as increasing the budget, because it makes sure that the new money goes into solving the real problems, not into management perks or bureaucratic empire building. The purpose of an organization is to get things done. To fulfill this purpose in a new technology organization which means making new things, the engineers must be supported by management. The engineers are the people who have to solve the problems. The proper place of management is to give them the tools and to fight for budget and priorities with upper management. Any other managerial function in an technology R&D organization that isn't concerned with sales and marketing is secondary at best and parasitic at worst.
Once upon a time, there was a political system whose management believed the country's problems could be solved by bureaucratic edict instead of with people finding out what the problems really were at an empirical level and solving them. The Soviet Union failed its reality check, just like NASA has repeatedly. The Soviet Union no longer exists. Perhaps it's time for NASA to follow it.
Space travel is dangerous. Live with the danger or get out of the business.
Ships were once dangerous. Automobile travel was once dangerous. Airplanes were once dangerous. Living in the America was once dangerous. Every new human domain has been paid for in blood. The problems were solved and now, kids can play outside in California suburbs without fear of being eaten by predators, they can fly in airliners without fear of following the trail of the Challenger astronauts.
The shuttle is not an example of how to deal with the dangers of space travel. Since it was designed, there have been 30 years of aerospace research and development. Can a new earth to LEO vehicle be designed with safety comparable to the DC-3? I think it's time to find out. Perhaps it can't be done, but we can't find out unless it's tried.
The DC-3 was a lot safer than anything that came before it. The modern jet airliner of today is a hell of a lot safer than the DC-3. It's called engineering progress, and that progress happens because engineers figure out what the problems are and their managers support them in getting the resources to implement the solutions. Not because PHMs attack them because they're saying things they don't want to hear.
Space travel is dangerous because Congress won't appropriate the funds to do what needs to be done to make it safe. This is largely because NASA management has not been able to make a case for it that Congress can understand. Even at the level of "if we don't, our astronauts will keep raining down on your constituents in barbecued chunks". Where is the engineering incompetence in this?
Where are the program directors with the integrity to say "We need this amount of money to put humans safely into space. If you won't give it to us, then you'll have to find other people willing to kill astronauts in order to give you guys good PR."
Either Congress should come up with the funds to develop a vehicle whose design takes into account what has been learned in the last 30 years or admit that America can't afford a real space program and leave the field to the private sector, the Indians, and the Chinese.
Tech Public Policy stuff