Shuttle Assessment Tool was Inferior

An anonymous reader writes " Shuttle report in Houston Chronicle: 'The computer program Boeing engineers used to predict that a debris-damaged Columbia could land safely wasn't much more than a simple chart of past foam damage, accident investigators said Tuesday.'"

Comments

The Tufte version

[RobotWisdom]

Edward Tufte is a demigod in the world of information-design, and he made an interesting case recently that bad PowerPoint design in Boeing's report contributed to the misinterpretation of the analysis. Eg, the way the ppt-slide was laid out almost completely concealed the fact that the test was on a small cube of foam.

Boeing's Analysis

[ASCIIMan]

For anyone that's interested in the actual Boeing presentation materials, NASA put copies up on their accident investigation website about a month and a half ago.

• January 21 Charts
• January 23 Charts
• January 24 Charts (warning: 1.3MB)

Oh, and here are some previous TPS Reports thrown in for good measure.

Feynman

[Henry V .009]

When I first heard of the foam analysis, it immediately reminded me of something that Feynman wrote in his Challenger report. This story seems to confirm the connection. I've typed out an excerpt from Feynman's report. It's worth reading. Feynman's brutalization of one of NASA's mathematical safety models in the third paragraph is the really relevant part.

The phenomenon of accepting for flight seals that had shown erosion and blow-by in previous flights is very clear. The Challenger flight is an excellent example. There are several references to flights that had gone before. The acceptance and success of these flights is taken as evidence of safety. But erosion and blow-by are not what the design expected. They are warnings that something is wrong. The equipment is not operating as expected, and therefore there is a danger that is can operate with even wider deviations in this unexpected and not thoroughly understood way. The fact that this danger did not lead to a catastrophe before is no guarantee that it will not the next time, unless it is completely understood. When playing Russian roulette the fact that the first shot got off safely is little comfort for the next. The origin and consequences of the erosion and blow-by were not understood. They did not occur equally on all flights and all joints; sometimes more, and sometimes less. Why not sometime, when whatever conditions determined it were right, still more, leading to catastrophe?

In spite of these variations from case to case, officials behaved as if they understood it, giving apparently logical arguments to each other often depending on the "success" of previous flights. For example, in determining if flight 51-L was safe to fly in the face of ring erosion in flight 51-C, it was noted that the erosion depth was only one-third of the radius. It had been noted in an experiment cutting the ring that cutting it as deep as one radius was necessary before the ring failed. Instead of being very concerned that variations of poorly understood conditions might reasonably create a deeper erosion this time, it was asserted, there was "a safety factor of three." This is a strange use of the engineer's term "safety factor." If a bridge is built to withstand a certain load without the beams permanently deforming, cracking, or breaking, it may be designed for the materials used to actually stand up under three times the load. This "safety factor" is to allow for uncertain excesses of load, or unknown extra loads, or weaknesses in the material that might have unexpected flaws, etc. If now the expected load comes on to the new bridge and a crack appears in a beam, this is a failure of the design. There was no safety factor at all; even though the bridge did not actually collapse because the crack only went one-third of the way through the beam. The O-rings of the Solid Rocket Boosters were not designed to erode. Erosion was a clue that something was wrong. Erosion was not something from which safety can be inferred.

There was no way, without full understanding, that one could have confidence that conditions the next time might not produce erosion three times more severe than the time before. Nevertheless, officials fooled themselves into thinking they had such understanding and confidence, in spite of the peculiar variations from case to case. A mathematical model was made to calculate erosion. This was a model based not on physical understanding but on empirical curve fitting. To be more detailed, it was supposed a stream of hot gas impinged on the O-ring material, and the heat was determined at the point of stagnation (so far, with reasonable physical, thermodynamic laws). But to determine how much rubber eroded it was assumed this depended only on this heat by a formula suggested by data on a similar material. A logarithmic plot suggeste3d a straight line, so it was supposed that the erosion varied as the .58 power of the heat, the .58 being determined by a nearest fit.