Shuttle Assessment Tool was Inferior

An anonymous reader writes " Shuttle report in Houston Chronicle: 'The computer program Boeing engineers used to predict that a debris-damaged Columbia could land safely wasn't much more than a simple chart of past foam damage, accident investigators said Tuesday.'"

Comments

The Tufte version

[RobotWisdom]

Edward Tufte is a demigod in the world of information-design, and he made an interesting case recently that bad PowerPoint design in Boeing's report contributed to the misinterpretation of the analysis. Eg, the way the ppt-slide was laid out almost completely concealed the fact that the test was on a small cube of foam.

Re:The Tufte version

[Tablizer]

"The reports provided the rationale for NASA officials to curtail further research (such as photographing the Columbia with spy cameras) on the tiles during the flight. Here is a close analysis of an important slide from a Boeing report. This discussion was prepared for a chapter on the cognitive style of PowerPoint in my new book Beautiful Evidence..."

It would be interesting to see the following headline in the NY Times:

"PowerPoint Killed Shuttle"

Yeah

[0x0d0a]

Shuttle report in Houston Chronicle: 'The computer program Boeing engineers used to predict that a debris-damaged Columbia could land safely wasn't much more than a simple chart of past foam damage, accident investigators said Tuesday.'

In other news, the Houston-based ContractorCorp announced its new, $20 million-dollar-a-license aerospace disaster analysis software...

Also today, President Bush vowed that "no cost would be spared" to identify that shuttle problem that "struck such a tragic blow to our nation's future"...

Excel spreadsheet

[Anonymous Coward]

I was listening to cspan last night and Gehman said this "program" was nothing more than an excel spreadsheet. And the chart of past damage simply could not predict damage from the size of foam that hit Columbia.

"Just a Spreadsheet" doesn't much matter

[0x69]

They had only guesses as to what kind of material (foam, ice, ice-loaded foam, etc.) hit the wing. Only crude estimates as to how much hit, & where. They'd NEVER done real inspections (ultrasound, X-ray, etc.) of those carbon-carbon composite leading edges (to look for delamination, fractures, internal erosion from oxygen entering through surface pinholes, etc.) I haven't heard that they had ANY real test data from larger hits.

In this context, it doesn't much matter whether the "program" is half a million lines of gigaflop-sucking Fortran or a Buck Rogers Secret Decoder Ring. They were (fairly contentedly) starved for meaningful input.

GIGO.

Re:"Just a Spreadsheet" doesn't much matter

[JUSTONEMORELATTE]

In this context, it doesn't much matter whether the "program" is half a million lines of gigaflop-sucking Fortran or a Buck Rogers Secret Decoder Ring. They were (fairly contentedly) starved for meaningful input.
You are absolutely correct in that they had no real data at the time. The problem is that they used an unsophisticated tool to perform a task for which was was never intended, and then (upon liking the results) stopped looking for more data or a better tool.
NASA had ordered telescopic images of the shuttle's wing. They cancelled that order when this "Buck Rogers Secret Decoder Ring" told them that everything was fine.
That is where the problem lies, and why "just a spreadsheet" matters.

--

Re:"Just a Spreadsheet" doesn't much matter

[The_K4]

Isn't hindsight a great thing? If they had known at the time that their analysis wasn't good enough do you really think they would have used only a spreadsheet? They much have thought that it was ok. Not to mention that even if they HAD said "it's dead" there would have been NO POSSIBLE WAY to save the crew. No matter how the numbers came out, once the foam hit the shuttle and caused the damage, it was naver comming home, and it's crew was going to suffer that fate with it. It was a tragic accident, let's LEARN from it, not point fingers and look down on people who did what they thought was the right thing!

Re:"Just a Spreadsheet" doesn't much matter

[ray-auch]

Not to mention that even if they HAD said "it's dead" there would have been NO POSSIBLE WAY to save the crew

Reports on released emails say that NASA engineers would seem to disagree. Maybe you're better at spaceflight than they are...

Re:"Just a Spreadsheet" doesn't much matter

[The_K4]

Notice the one thing missing from that artice, a clear, workable plan. Even if they HAD decided to rescue them:
1) There were no other shuttles ready to launch, and none could have been preped and launched in time
2) The only other space agency in the world that had put a man in space and returned him safly is the russians. They COULD have had a launch ready in time, however they can carry only 3 people back. They would have needed at least 2, probably 3 to make it work. That they couldn't have done. Note: They coulve have launched 1 and then moved the rest of the crew to the ISS and used the life boat, AND left 1 extra person on the ISS. However then you have 4 people on the ISS which is rigged for 1 - 3 and NO life boat and it would be several months till that was replaced. Oh yeah, the shuttle was also in the wrong orbit to dock with the ISS, but that MIGHT have been ok if they had enought fuel.
3) The shuttle can't dock with a russina craft with out the arm, which wasn't on board, it would hve been VERY difficult for them to get 7 people from the shuttle ANYWERE with out docking (if you miss where your going space can be a long slow death)
4) They CAN NOT have repaired the shuttle.

Now I admit that IF NASA had decided that the shuttle was dead and started working with what they had they MIGHT have come up with a workable plan, NASA's got a lot of really smart people. However that article is by now means a blue print for saving them. I should ahve been more accurate and said NO KNOWN WAY to save them, because it might have been possible, we'll never know.

Re:"Just a Spreadsheet" doesn't much matter

[ray-auch]

We already know your 1-4 were not options, so did NASA, which is why they weren't discussing them. Also, the shuttle didn't have enough deltaV to reach the ISS so that wasn't an option anyway.

The point is that while the media and other commentators were saying "there was nothing they could have done", the NASA guys had discussed things they could have done, had the damage been thought serious enough.

The mindset that got Apollo 13 back doesn't seem to have disappeared, as some have suggested, it was just not thought necessary to apply it.

By the way, you missed out the common 5), which is "they couldn't have changed the reentry angle because then the shuttle would have been going too fast to land". So how come they were considering changing re-entry trajectory then ? - think about it.

Re:"Just a Spreadsheet" doesn't much matter

[The_K4]

They can't change it too much, because they don't want to bounce off the atmosphere. You go too steep you burn up, to shallow and you bounce off. :) Also the shuttle can't "fly" it's like a brick with really stubby wings, even assuming that you could come down in such a way that heat wasn't the problem, you still need to be able to land.

I do see your point. There may have been SOME options, but with the clock ticking, and very few materials to work with, it probably wouldn't have saved them. :(

Re:"Just a Spreadsheet" doesn't much matter

[ray-auch]

[...]you still need to be able to land.

Not so - if it is decided the shuttle won't make landing then you can go for bail-out instead. This was discussed in one of the released NASA emails in the context of landing-gear door burn-through leading to gear failure, and needing to decide on belly-landing vs. bail-out.

The re-entry trajectories are calculated with the target of the glide path to landing. If you know (or think you know) that those trajectories will result in loss of a wing, then you aren't landing.

In that case, the glide path target is invalid, and instead you need to hit flight regime suitable for bail-out, preferably with the vehicle impacting somewhere uninhabited. This changes the assumptions under which the re-entry trajectories were calculated, so now you need to go back and do them again - with the less strict end-point hopefully meaning far more options higher up.

I admit it is all pretty theoretical - to get to this point they would have to have known, in orbit, that the damage was serious enough that the shuttle was lost and they were just trying to save crew. Given the damage models they were using for the foam impact, I don't think it is likely they have good models for consequences of leading-edge damage in re-entry (we're assuming they knew the damage was there), so I guess they would have tried for a landing trajectory anyway.

If they lose a chunk of leading-edge on a future flight, then I expect they'll be looking hard at options like this.

What choice did they have?

[GypC]

Seems like a moot point to me. From what I understand they had no alternative but to attempt a landing. Maybe if they had somehow scraped together another shuttle launch right after the first one they could have all ridden home in the second one? Or maybe fixed the damage to the first one? I doubt it.

Re:What choice did they have?

[Smallpond]

Yeah. Just like the way they gave up on Apollo 13 after it exploded in space. Oh wait, they got it back safely.

Don't for a second think that NASA engineers take their job lightly. The ones I've known are very committed. Look at the number of emails and memos that were passed around about the foam hitting the wing.

Hindsight says they should have taken the warnings more seriously, but they get warnings on every flight. If they grounded the shuttle until it was as safe as a passenger plane, we just wouldn't be in space at all.

Re:What choice did they have?

[sl3xd]

Just like the way they gave up on Apollo 13 after it exploded in space. Oh wait, they got it back safely.

That's a pretty cynical attitude. It also greatly underscores the fact that we nearly lost Apollo 13's crew, for several reasons. In fact, Houston had every reason to expect that Odyssey would break up during re-entry as well; it is a miracle that it didn't. In fact, NASA did as much giving up then as they did with the shuttle. In both cases, it was unrealistic to 'mount a rescue'; spaceflight isn't that commonplace, which is a fact many seem to forget. Launching payloads into orbit isn't a daily or weekly thing; manned spaceflight is even less common; maybe a monthly thing, if that. NASA basically had the choice: Re-enter now, and take our chances, or re-enter later, and take the same chance. With Apollo 13, NASA gave up the moment they told the crew of the Odyssey to do the final retro-burn to re-enter Earth's atmosphere. They had no choice -- if they didn't bring Odyssey down, the crew would die of asphyxiation, or the extreme temperatures of space.

The same choice presented itself with Colombia; a 'rescue' is only slightly more plausable now than it was during Apollo; which is to say, one step above utterly implausable.

The fact of the matter is that in both cases, the craft had to come down if the crew were to have any chance of survival.

For Columbia, docking with Station Alpha was impossible-- wrong orbit, not enough fuel. There was an oxygen reserve for a few days to a week, depending on who you talk to. It's a moot point, since those few days are insufficient to launch any kind of rescue; it takes weeks to get a scheduled shuttle launch going; hell, it takes weeks to get any rocket ready for launch. And it's not like Boeing, Lockheed, Arianne, or Russia have a spare launch vehicle laying around prepped and ready to go for a rainy day; they certainly don't have two, which is the number of Soyuz craft it would take to return the crew back to Earth. It costs serious $$$ to keep a rocket in a 'prepped and ready to go' state; enough so to make it impractical.

There have been many, many cases where the heat shields of a spacecraft were damaged, or uncertain: Friendship 7, Apollo 13, a couple of the Gemini missions, and at least 20% of the shuttle flights. (There were entirely missing tiles when Columbia made its maiden voyage, and this has repeated itself several times on every one of the shuttles that have flown.). All of them turned out well.

So, there was a choice: Die slowly of asphyxiation and/or dehydration, hoping that the (extremely long) odds of survival until a 'rescue' could be mounted would favor you, or take the much more comfortable odds that you will die during re-entry, when death would take a few microseconds?

No matter what option was taken, the crew would still have to go through re-entry; the only difference would be the craft it happens with.

Re:What choice did they have?

[gizmo_mathboy]

...For Columbia, docking with Station Alpha was impossible-- wrong orbit, not enough fuel.

How about they couldn't dock because they didn't have the correct docking mechanism.

If they had enough fuel to get to the Space Station(SS) about the only choice they had to get the crew onto it was to use the 2 space suits to ferry folks from Columbia to the SS. I'm assuming that there is an airlock on the SS that can be used for EVA's.

This might have given folks enough time on the ground to get 4 (need a trained pilot for the return I'm guessing) Soyuz capsules up there and rescure the Columbia crew.

Of course, this crazy scheme depends on there being enough fuel onboard Columbia to get it close to the SS.

I just hope we truly learn from this mistake.

Re:What choice did they have?

[YrWrstNtmr]

Or, if they had known about the situation earlier (inspection via telescope) they could have come up with something.

It was a 10 day flight. Knowing about the problem a couple of days into it would have given MUCH better options, rather than just winging it on the reentry.

Stretch the rations/air for a few more days. "hey...there's a Progress resupply going to the ISS next week. Can we divert that and give them another week or so?"

Who knows? But I do know that having 2 weeks to figure something out is preferable to having 30 seconds.

Whatever happened to Failure is not an option.

Re:What choice did they have?

[sl3xd]

Or, if they had known about the situation earlier (inspection via telescope) they could have come up with something.

How much do you not know about optics? That argument is a fairly bad joke, in my view. There aren't many spy-sattelites sitting on the ground, pointing upward -- espescially ones on a motor-driven mount that is smooth enough to provide a clear picture, is there? And don't give me anything about astronomical telescopes, which are the only other thing that can see clearly enough -- there is a *big* difference in the slew speed between keeping a star targeted, and keeping an orbital spacecraft targeted. They couldn't move around fast enough to do any good. And more importantly, their pictures are time-exposures, taking several minutes to take. The shuttle passes from one end of the horizon to the other in less time than the observatory's targeting systems can precicely move. Plus, the ground-based telescopes that are powerful enough to see any damage are too 'far-sighted' to see the shuttle anyway.

And spy sattelites in orbit? Well, there are a couple of problems: Wrong orbit; the cameras in them are designed to look at the ground. Its control algorithms are carefully tailored to lock on a target on the ground, which gives a fairly steady and easy-to follow heading. The shuttle is moving in a skewed direction, at a *much* higher speed than the camera is designed for. So much for a nice clear picture... There's also the issue of looking at the black bottom of the shuttle; unless, somehow, you got the perfectly ideal shot-- the one in ten billion odds that you hit the CCD when you get a reflection from the sun off of the tiles, you dont' see anything but black on black. Re-positioning the shuttle can try to even the odds up a bit; but it's a long shot at best.

And even if they could see damage severe enough to be a problem (which is unlikely, a crack in the tile could quite easily be invisible anyway)

You can only come up with a solution if you have the tools and materials to make it work. The shuttle does have some tools on it; but as they didn't have as part of their mission anything that would require the necessary tools, they didn't go up with the shuttle every pound launched costs a great deal of $$$. (why pay 13,000+ extra for a screwdriver the mission does not call for?) But even assuming that they did have a full toolbox (which is the safer assumption), it takes ground crews weeks to get the tiles into place on earth. (Although this is for the 'entire' set of tiles).

In space, it is much, much, much harder. First, you have to remove the damaged tile, which fits so closely to the others that it would require some kind of drill/screw to pull it out without damaging the other tiles, or you're just as dead. There's the issue of whether a human is strong enough to actually do this (they are structurally very strong, and even with a crowbar it's doubtful that human muscles could pry a tile up at all -- espescially in zero G where you can't 'put your weight into it' You have to push the prybar 'down' to force the tile 'up'. Only there is nothing to push 'down' with -- just empty space. So, then you wrap some kind of rope loop around the entire fuselage (several hundred feet worth, or it would slip off -- it has to go around the back of the shuttle as well). Then there may be something with which the spacewalker could apply the proper force required to pry up the tile. But then the rope stretches; stretching the rope taught enough to provide enough force to pry up the tile would require a ratcheted pulley (not unrealistic), and the force it would apply on the areas of the shuttle it chafes with would be enough to crack even more tiles -- and in places that it is even worse to pry up (like the leading edges of the wings).

Keep in mind, these tiles are tough; they have to be able to withstand paint flecks colliding with the shuttle with an impact speed of well over 50,000 mph. The tiles are regularly tested for

Re:What choice did they have?

[Tablizer]

Seems like a moot point to me. From what I understand they had no alternative but to attempt a landing.

That is not quite true. There are some suggestions that later popped up that could have reduced the chance of outright failure.

One was to turn the damaged section away from the sun for a few hours to cool it down a bit.

Second, they could eject some non-essential items to make the shuttle lighter, reducing re-entry drag-time.

Re:What choice did they have?

[Tablizer]

And don't give me anything about astronomical telescopes, which are the only other thing that can see clearly enough -- there is a *big* difference in the slew speed between keeping a star targeted, and keeping an orbital spacecraft targeted.

But they *did* take an image while it was up there. It did not show the area of concern because the shuttle was facing the wrong direction at the time, but it looked clear enough to be able to show a dark hole about a foot or so in diameter to me, perhaps even 6 inches if the lighting was right.

Nasa did not authorize it, it was just an observatory "playing around" IIRC. Does anybody have a link to the photo? It is one of those historical "Big Almost's" if you ask me, on par with the evaluation form of the recording executive who rejected the Beatles because "guitars are on their way out".

Re:What choice did they have?

[Tablizer]

But they *did* take an image while it was up there.

I found a link to such an image

Re:What choice did they have?

[YrWrstNtmr]

Or, if they had known about the situation earlier (inspection via telescope) they could have come up with something.

How much do you not know about optics? That argument is a fairly bad joke, in my view.

Not to dredge up an old post, but apparently someone on the CAIB thinks we can and should do it.

"Recommendation Two: Prior to return to flight, NASA should modify its Memorandum of Agreement with National Imagery and Mapping Agency (NIMA) to make on-orbit imaging for each Shuttle flight a standard requirement.

This recommendation was issued because of the board's finding that the full capabilities of the United States Government to image the Shuttle on orbit were not utilized."

Re:What choice did they have?

[sl3xd]

Not that any such images would have changed the fact that there simply was no other way to bring Columbia down. At least, not with a living crew.

Boeing's Analysis

[ASCIIMan]

For anyone that's interested in the actual Boeing presentation materials, NASA put copies up on their accident investigation website about a month and a half ago.

• January 21 Charts
• January 23 Charts
• January 24 Charts (warning: 1.3MB)

Oh, and here are some previous TPS Reports thrown in for good measure.

Feynman

[Henry V .009]

When I first heard of the foam analysis, it immediately reminded me of something that Feynman wrote in his Challenger report. This story seems to confirm the connection. I've typed out an excerpt from Feynman's report. It's worth reading. Feynman's brutalization of one of NASA's mathematical safety models in the third paragraph is the really relevant part.

The phenomenon of accepting for flight seals that had shown erosion and blow-by in previous flights is very clear. The Challenger flight is an excellent example. There are several references to flights that had gone before. The acceptance and success of these flights is taken as evidence of safety. But erosion and blow-by are not what the design expected. They are warnings that something is wrong. The equipment is not operating as expected, and therefore there is a danger that is can operate with even wider deviations in this unexpected and not thoroughly understood way. The fact that this danger did not lead to a catastrophe before is no guarantee that it will not the next time, unless it is completely understood. When playing Russian roulette the fact that the first shot got off safely is little comfort for the next. The origin and consequences of the erosion and blow-by were not understood. They did not occur equally on all flights and all joints; sometimes more, and sometimes less. Why not sometime, when whatever conditions determined it were right, still more, leading to catastrophe?

In spite of these variations from case to case, officials behaved as if they understood it, giving apparently logical arguments to each other often depending on the "success" of previous flights. For example, in determining if flight 51-L was safe to fly in the face of ring erosion in flight 51-C, it was noted that the erosion depth was only one-third of the radius. It had been noted in an experiment cutting the ring that cutting it as deep as one radius was necessary before the ring failed. Instead of being very concerned that variations of poorly understood conditions might reasonably create a deeper erosion this time, it was asserted, there was "a safety factor of three." This is a strange use of the engineer's term "safety factor." If a bridge is built to withstand a certain load without the beams permanently deforming, cracking, or breaking, it may be designed for the materials used to actually stand up under three times the load. This "safety factor" is to allow for uncertain excesses of load, or unknown extra loads, or weaknesses in the material that might have unexpected flaws, etc. If now the expected load comes on to the new bridge and a crack appears in a beam, this is a failure of the design. There was no safety factor at all; even though the bridge did not actually collapse because the crack only went one-third of the way through the beam. The O-rings of the Solid Rocket Boosters were not designed to erode. Erosion was a clue that something was wrong. Erosion was not something from which safety can be inferred.

There was no way, without full understanding, that one could have confidence that conditions the next time might not produce erosion three times more severe than the time before. Nevertheless, officials fooled themselves into thinking they had such understanding and confidence, in spite of the peculiar variations from case to case. A mathematical model was made to calculate erosion. This was a model based not on physical understanding but on empirical curve fitting. To be more detailed, it was supposed a stream of hot gas impinged on the O-ring material, and the heat was determined at the point of stagnation (so far, with reasonable physical, thermodynamic laws). But to determine how much rubber eroded it was assumed this depended only on this heat by a formula suggested by data on a similar material. A logarithmic plot suggeste3d a straight line, so it was supposed that the erosion varied as the .58 power of the heat, the .58 being determined by a nearest fit.

Economics in a country that demands Profit!

[purduephotog]

First, I am an engineer. Alot of you probably are too. What I'm going to say will probably be modded flamebait, etc, but I'm fighting my own battles at work in regards to problems that no one else saw... or care
Alot of the analysis has been attacking the engineers for not asking enough questions. Thats fine and dandy in a 100% hindsight problem- we have a failed shuttle- lets' find out why. Alot of the reviews have been talking about data presentation- thats good too- I went to school for engineering, not marketing, and therefor don't know what a marketer does as to how to present information without getting bogged down in details.
But when it comes straight down to it, it's money, pure and simple. Do you think CAT scans of tiles are inexpenisve? Probably a couple $k each. Do this for every tile. Want to understand turbulence completely (and people that say you can model a chaotic system- just watch the weather channel to know how EASY that is)- that costs money and time. Quite a bit of both, too.
So now you've got budget concerns on projects that aren't funded and you can only skunk work it too much (note- skunk work is done on the side, unpaid overtime/salary, and 'hiding' the cost of equipment time/usage under a variety of things. It's amazing what you can do sometimes).
Now and then you get lucky and management comes around... funds your project, everyone gets paid with a little back in the jar for the next skunk project... then again, what does management usually know? zip. Just those bottom line numbers
Now obviously there was a bit of scaleup issue. I'm not comfortable with a 5x scaleup on some jobs, much less a 640x prediction- thats me personally. And the analysis that reads safety as a failure, instead of safety as a problem is dead on (1/3 the O-ring). But don't go too hard on the engineers- many comments are headed that way. Just remember under-funding answers the important questions, and may lop a bunch of details under assumptions... and every now and then you get bit in the ass... hard.

Re:BOEING company is a piece of shit...

[dnahelix]

Modding me down still doesn't change the fact that

PHIL CONDIT SUCKS DISEASED DONKEY DICK!!