Slashdot Mirror
Failure Is Always an Option
Logic Bomb writes "The New York Times has a short but elegant op-ed regarding the different perspectives of engineers and managers and the role that plays in accidents like the space shuttle Columbia disaster. It's the sort of article you'll nod all the way through, then print and leave anonymously on your supervisor's desk. Any tech managers in the Slashdot crowd might have some interesting comments on how the right balance is struck." Henry Petroski has written several good books on engineering and failure.
In the case of my last software project, engineers who worried about bugs that the software may have suffered during design were ineffective in getting it properly inspected before launch.
"No boss, I have no idea where that article printed out 15 times and strewn across your office came from........ It looks like a good article though."
"I can not bring myself to believe that if knowledge presents danger, the solution is ignorance" - Isaac Asimov
By Homer Hickam
When I go to the Cape and watch the Shuttle being launched, I still get a lump in my throat watching it soar. Even though I no longer work for NASA, its thunder affirms my dreams for spaceflight. Still, when I put emotion aside, I can't ignore my engineering training. That training and my knowledge as a 20-year veteran of the space agency (and also a Vietnam vet) has led me to conclude that the Space Shuttle is NASA's Vietnam. A generation of engineers and managers have exhausted themselves trying to make it work and they just can't. Why not? Because the Shuttle's engineering design, just as Vietnam's political design, is inherently flawed.
Much has been made of the report produced by the Columbia Accident Investigation Board (CAIB). I've read newspaper articles that called it "scathing." Hardly. Its tepid recommendations probably had Shuttle managers who made poor decisions dancing with relief. It gave them a pass by proclaiming "culture" made them do it.
I don't believe there's a NASA culture. There is, however, a Shuttle cult. It is practiced like a religion by space policy makers who simply cannot imagine an American space agency without the Shuttle. Well, I can, and it's a space agency which can actually fly people and cargoes into orbit without everybody involved being terrified of imminent destruction every time there's lift-off. With some reservations, written in the politest language, the CAIB recommended to keep Shuttles flying but with more inspections, more bureaucracy (an outside safety agency), and more money. But piling on more inspections, people and dollars won't make the Shuttle safer. Neither will the safety sensitivity training that will probably be dumped on top of the overworked, disillusioned NASA engineers. My God, they've already dedicated their very souls to keep the Shuttle flying safely! The truth is, no amount of arm-waving about "culture" can fix a flawed design.
Take a look at the Shuttle stack and what do you see? A fragile spaceplane sitting on the back of a huge propellant tank between two massive solid rocket boosters. The Shuttle has to sit right in the middle of all the turmoil of launch because we once believed it would be cheaper to bring back those engines and rebuild them than to build new ones. That has not proved to be the case -- far from it -- but it has left us with a crew sitting in the most vulnerable position possible in terms of design. Simply put, had that spaceplane been on top of the stack, the destruction of Challenger and Columbia wouldn't have occurred. The CAIB ignored this flawed design and that makes their conclusions suspect: no amount of inspections or condemning another NASA generation to worry over this thing will solve it.
So let's get practical. We can't just shut the thing down. We need the Shuttle to finish the space station and also to keep the Russians and Chinese from dominating space. I'm not willing to see that occur while we dither. Human spaceflight is important to this country. But the Shuttle is as safe as you're going to get with what's in place today. Let's put some tough engineers in charge, fly it 10 more times over the next four years with hand-picked crews to finish the space station and meet our international obligations. Then close the program and replace it with expendable launchers and a shiny new spaceplane. And, this time, put it on top.
Was it Thomas Edison that said, "I haven't failed. I just found 10,000 ways that didn't work."?
"It's better to have a gun and not need it than need a gun and not have it." ~ Christian Slater, True Romance
Now that money may be in the form of lower gas mileage in a car, or in the form of hundreds of unmanned test flights before putting a human in, or obscene safety margins.
But to pretend that anything is ever perfectly safe is to ignore the fundamental economic issue that at some point you have to stop putting money into safety concerns and just fly the damn thing.
On a project the size of the space shuttle thousands of safety concerns will be brought up. Not everyone of them can be fully investigated. They have to pick and choose based on what is most urgent. Yes, there will be accidents, but otherwise the shuttle would never get off the ground. Hindsight is twenty-twenty, and you can say they should have investigated further all you want, but the fact is that there were many other concerns that seemed just as urgent, and some that seemed even moreso.
Slashdotter are stupid and biased.
Failure Is Always an Option
By HENRY PETROSKI
URHAM, N.C. -- Scientists seek to understand what is, the aerospace pioneer Theodore von Karman is supposed to have said, while engineers seek to create what never was. The space shuttle was designed, at least in part, to broaden our knowledge of the universe. To scientists the vehicle was a tool; to engineers it was their creation.
With the release of the report of the Columbia Accident Investigation Board, there is a new focus on the "culture" of NASA. Engineers have played a prominent but not a controlling role in that culture, both in the design of the shuttle and in the planning of its missions. When the report speaks of NASA's "broken safety culture," the particular failure it cites is "a consistent lack of concern" that Columbia may have been damaged by debris at takeoff. But perhaps NASA can be better understood by examining the culture that arises from the inevitable -- and healthy -- tension among scientists, managers and engineers.
A common misconception about how things such as space shuttles come to be is that engineers simply apply the theories and equations of science. But this cannot be done until the new thing-to-be is conceived in the engineer's mind's eye. Rather than following from science, engineered things lead it. The steam engine was developed before thermodynamics, and flying machines before aerodynamics. The sciences were invented to explain the accomplishments -- and to analyze their shortcomings.
The design of any device, machine or system is fraught with failure. Indeed, the way engineers achieve success in their designs is by imagining how they might fail. If gases escaping from a booster rocket can lower efficiency or cause damage, then O-ring seals are added. If the friction of re-entry can melt a spacecraft, then a heat shield is devised.
Much of design is thus defensive engineering: containing, shielding and fending off anticipated problems on the drawing board and computer screen so that they cannot bring down the design when it flies. Obviously, total success can only come if every possible mode of failure is identified and defended against.
Engineering is also very much about numbers. O-rings must be sized; the thickness of heat shields specified; the weight of insulation calculated. Often, the numbers work at cross purposes, as when increasing shield material decreases available payload. Engineering design is ultimately the art of compromise.
What results from the design process is a thing that has unique characteristics. It can withstand the conditions for which it was designed as long as it maintains its integrity. There is usually some leeway allowed, for engineers know that operating conditions cannot be predicted with absolute certainty. Until it fails, how far beyond design conditions a system can be pushed is never fully known.
But engineers do know that nothing is perfect, including themselves. As careful and extensive as their calculations might be, engineers know that they can err -- and that things can behave differently out of the laboratory. On the space shuttles, O-rings got scorched, heat tiles fell off, foam insulation broke free. To engineers, these unexpected events were incontrovertible evidence that they did not fully understand the machine.
Engineers do not feel comfortable with things they do not understand. It is at this point that they begin to act more like scientists. In the case of the scorched O-rings, the engineers studied burn patterns. They looked for a correlation between damage and temperature, and they warned about launching when the temperature was outside the bounds of their experience and scientific study.
If engineers are pessimists, managers are optimists about technology. Successful, albeit flawed missions indicated to them not a weak but a robust machine. When engineers and managers clashed over the 1986 Challenger launch, the managers pulled rank. In the case of Columbia, engineers who worried about damage that the
The problem is that people are afraid that if the shuttle stops flying space exploration will stop. Public support will wane and funcing will slow. I happen to disagree but there are many in the space program who do not.
"I can not bring myself to believe that if knowledge presents danger, the solution is ignorance" - Isaac Asimov
After you make such significant strides in space exploration in the late 60's and early 70's, then have your funding cut by almost every President since Nixon, you're bound to start taking short-cuts and missing things. Remember... space is still deadly. In my book, when you're dealing with something that could very easily kill you, you don't short-change yourself. The problem is that when you have no money to spend on things you need, and a time limit to do certain things, you don't have any other choice.
The problem NASA has right now is trying to convince the rest of the country that what they're trying to do is worth spending the money on. Why worry about what Saddam can do if we could all just move to Mars (for instance)? On the other hand, funding was cut because nothing significant was happening... but nothing significant was happening because funding was cut. It's a vicious cycle.
"It's better to have a gun and not need it than need a gun and not have it." ~ Christian Slater, True Romance
I have spent the last few days reading the entire CAIB report and I have to agree that Mr. Petroski is right on target with his observations.
Simply put, the problem was that the engineers concerned with the safe re-entry of the orbiter after the foam strike were put in the position of having to prove a negative. Management wouldn't pay attention to them until they could prove that the strike was *not* safe.
They couldn't prove or disprove the notion that the foam strike had caused critical damage until they got the images, but they couldn't get the images without first proving they needed them to assure the safety of the re-entry.
There had been a number of previous foam strikes, many of them involving this same piece of foam (the left bipod ramp), and all of those shuttles had landed okay, so management believed that this foam strike was similarly okay just because they had gotten away with it so far.
No science. No analysis. Just an assumption that if they had gotten away with ignoring this problem so far, they could continue to ignore it. The schedule was king, not safety.
Engineers know well that "getting away with it" is not evidence of reliability. Managers, at least in my experience, tend to be proportionately successful in their careers to the extent that they can spin "getting away with it" into a career advancement tool.
This is really why the orbiter was lost. This is really why the astronauts died.
Denial is deadly.
The best way to do is to be.
I have to admit up front that I am biased against NASA on primarily ethical grounds. To me, there's one basic valid purpose of government, and that's to defend the individual rights of its citizens. In the U.S., this is the principle upon which the Constitution and Bill of Rights is based, and the primary ligitimate activities of government, the police, courts, and defense, are inferrable from that.
Everything has an opportunity cost. The money spent on NASA could otherwise be spent elsewhere, such as aiding the homeless or better road infrastructure, and preferably on something the person earning the money (the taxpayer) himself chose.
Sure, it's nice to be able to explore space and determine facts about physics and cosmology, and test theories against empirical information, but I think at some point the costs associated with expanding the realm of science to more obscure areas, in the shorter term, are too high. And, yes, I know the argument that expanding basic science can lead to invention that benefits the individual, but personally I'd put more faith in the ability of industry to use the money making targeted investments while hiring scientists, than effective production from NASA. At some point I think we have to say the money can be better spent than knowing more about the behavior of some unreachable binary star. Eventually, that information will likely come anyway, as a function of better theoretical models. Why do we need it now, assuming it isn't primarily to give a Ph.D. something to play with?
NASA exists in an enviroment that offers none of the efficiency advantages of modern industry.
- No effective competition
- No way to inexpensively prototype or proof-of-concept things and test them in the intended deployment environment
- Few efficiencies of scale from being able to buy parts widely used and commoditized
- Little economic justification for the expense, even in the instances where the mission is "successful"
- No realistic, market-driven benchmarks for the performance of the managers or engineers
In the end, I don't feel that NASA is an optimal way to spend money, and since it's at least in part my money, I should be able to make this decision. Perhaps some kind of opt-in "NASA" checkbox, like I've seen opt-in "environmental" checkboxes on tax forms. I'd be content with that.
~ Whence do you come, slayer of men, or where are you going, conqueror of space?
This is the same Homer Hickham about whom October Sky was made, I'm assuming?
It would be nice if more people listened to engineers instead of politicians when it came to science projects, wouldn't it?
3D Printing Tips and Tricks at Zheng3.com
I opened this at work, and the title bar reads:
"Failure is always an option - Microsoft Internet Explorer"
Gotta love it!
Acts of massive stupidity are almost never covered by warranty. --me.
This is always the case it has been for a very a long time. The problem is not NASA's culture so much as the culture of the society around NASA.
The article Misses the big points. When the Challenger blew up blame was apportioned to the engineers that built it not the congressmen who insisted the engines be built in utah. When software is shipped before its ready, blame goes to the programmers that were working 90 hour weeks not the sales people that promised the customer whatever they wanted to hear. When a heartvalve fails blame goes to the inventors that made a device that saved lives, not the insurance companies that wouldnt pay for a proper solution.
Yes managers are willing to take risks, its rare they ever have to pay the price for failure.
I work for an auto supplier. In one of the prototype plants, there was a banner for one of the new car's engineering team.
"Failure is NOT and option."
It struck me as odd at the time. It just doesn't sound like motivation. It strikes me as a negative way of looking at things. There was no "We can succeed together!" or "Hard work will pay off in the end!" Nope. Failure is not an option.
Later I saw the perfect response in a magazine, and was disappointed that the banner was taken down before I could add it.
"Failure is not an option; it comes standard with every vehicle."
"...At the end of the day"..."when everyone goes home, you're stuck with yourself." RIP Layne Staley
Sadly, many higher-ups see the solution in CMM, or other quality programs that produce reams of paper, but those same top-level managers ignore the economics of trying to develop too much, in too little time, with too little money. I manage the development of custom software projects for a Fortune 100 company, and at the end of the day, the sales dweeb sells whatever he has to to make his commission, and the engineering group is left with impossible constraints. CMM would probably work well if the entire company bought into it, but I've not seen that yet.
Likewise, NASA sees us (the public) crying about cost overruns and the return on our investment. Ultimately, that comes back down to the line-level managers at NASA, where no matter what the good intentions, the pressures of $$$ and time will always apply.
We need to decide if space travel is worth the cost (done properly, and left to engineering minds to decide what "properly" means), or worth the risks of doing it at lower cost. Like my company, NASA has squeaked by on luck for quite some time.
In my experience, the luck ALWAYS runs out.
Tim
It's worth thinking about what would have happened if the damaged Shuttle had been images by USAF ground cameras, and it became clear that re-entry was going to be a disaster. The shuttle and crew would have been stuck in orbit, with worldwide publicity, while NASA tried to come up with a fix. They probably wouldn't have succeeded. On-orbit rescue using Atlantis has been discussed as marginally possible, and on-orbit patching has been suggested, but most likely, they wouldn't have worked.
Think of the PR fallout. Seven astronauts stuck in orbit for most of a month, with constant TV coverage, followed by their deaths on worldwide TV. That would have been career-ending for most of NASA's top management. Letting them crash saved the jobs of top people at NASA.
Worst case, a rushed launch of Atlantis could have resulted in losing two shuttles. That would have ended the Shuttle program.
Hickam is on track, but I'm not sure we need spacecraft with wings. Wings are only useful on airplanes. By definition, spacecraft are not airplanes. NASA has thrown away too much money pursuing winged spacecraft for their own sake, rather than dealing with the issue of getting people to and from space. They might as well try to make a submarine that can fly. Probably do-able, but: why?
Let's decide that we will do two things:
1) Any human space travel beyond LEO will start from LEO in spacecraft built in LEO and that return to LEO. If we do that, we will never need to spend money trying to build airplane-spacecraft hybrids.
2) Let's use big expendable boosters to get hardware to LEO, and smaller expendable boosters to get people to LEO. Put the people in modern versions of the Apollo or Gemini craft (the so-called "Big" Gemini was an appropos solution)>
And, let's also decide that the main reason to build a space station in LEO is to serve as a construction yard and a gas station for trips elsewhere. Let's put aside the quaint notion that the reason we need to be in space is to "do science".
-- Slashdot: When Public Access TV Says "No"
It's a sober and informed discussion of engineering safety (mostly but not entirely computer related) that's been going on for almost twenty years.
Try entering "shuttle" in the search form. I did just now and found the brief, grim announcement of the Challenger explosion.
If you prefer to curl up with a dead tree by the fire, read moderator Peter Neumann's Computer Related Risks. It is also available in Japanese translation.
Now, few of us are likely to ever risk our lives flying in space shuttles. Maybe some of us might write the code or design the machinery the astronauts will trust with their lives. But all of us depend on computers every day for our livelihood, and many of us depend on them for our lives more than you would feel comfortable with if you understand the implications of it.
Fly on an airplane lately? Anything a little more modern than a DC-3? Do you know what fly by wire means? Ever write code with a stack overflow or heap corruption? What do you suppose that means for the embedded systems that run today's commercial aircraft?
Does your car have antilock brakes?
Read RISKS. It will make you a better programmer. Because it will put the fear of God into you.
Request your free CD of my piano music.
I've been involved in engineering literally all my life. My dad was an engineer and as a small child I remember going to work with my dad and being in awe of all the stuff he had to 'play' with. I never wanted to be anything else! Unfortunately, in the scheme of things we are the workers, the ones who toil withput credit. The managers take all of that. In the 1980's as a contract engineer I built a Boston FM radio station from scratch (WFNX), yet they didn't even see fit to invite me to its sign on party! When I asked why, I was told: "You were paid well for your work, isn't that enough?". They actually believed they paid me too much to make their property worth many millions morethan it was before. Needless to say from that time forward, I did only precicely what they paid me to do (and what they asked me to do), nothing more. Part of the problem is we ALLOW ourselves to be treated in this way! The plumber, electrician or auto mechanic don't. Why do we? I think one answer is UNION. They realize there is respect and safety in numbers. Are we too good, too elite to do the same?
NASA isn't getting criticized because it doesn't have perfect safety, it's getting nailed because it has TWICE ignored clear evidence of significant problems and failed to perform even cursory investigations until after the loss of an orbiter and crew.
There was clear evidence of problems with the O-rings before the Challenger was lost. NASA had somebody produce some really cryptic plots, but nobody bothered to really investigate whether the cooler weather on some of these launches might have an influence. It takes a real genius to reduce this to dipping an o-ring into a glass of ice water, but any competent investigator should have been able to reduce the data to plots of damage vs. various independent variables such as temperature at launch or overnight lows.
With Columbia, the arrogance of management is far more stunning. It KNEW that the insulation had flaked off, it KNEW that the insulation had caused surface damage in the past, and it KNEW that some areas on the leading edge of the wing are much more vulnerable to damage than others because of access points. It could have test fired foam at wing mockups at any time, just to have hard proof instead of just hunches that the foam could never cause significant damage to an orbiter... yet it did nothing.
This testing is expensive, of course, but it's really not that much when compared to the cost of a normal launch (isn't that approaching a billion dollars per launch now?), or the various costs associated with the loss of an orbiter and crew. It's akin to failing to spend $10 to check something on your car even though you knew that a mistake would mean that the car would erupt into a fireball and kill everyone inside if you're wrong.
For every complex problem there is an answer that is clear, simple, and wrong. -- H L Mencken
When engineers and managers clashed over the 1986 Challenger launch, the managers pulled rank.
What a dark, yet utterly true statement. Do the NASA and contracting company managers sleep well today knowing that in 1986 their decisions cost lives?
Edward Tufte, author of some amazing books on information display, wrote in Envisioning Information on the Challenger disaster. Looking at the materials prepared by engineers, he saw that they had correctly correlated temperature with O-ring failure. Yet their materials, hastily prepared during the 11th hour, failed to convince managers to abort the launch. Tufte shows a design of a simple graph that shows temperature on the abscissa and burn-through on the ordinate, and any manager could draw a line through the points and extrapolate out to the bitter cold Florida day that cost the shuttle.
Having my own share of bad managers, I have to wonder, would it have made any difference?
But try as I might, I can't lay 100% of the blame on them: they see the budget for aero and space research being cut (more tax-cuts for the wealthy!!!) and they know they need to get public opinion behind them. That means the Shuttle must fly, and it must be a media spectacle.
The truth of the matter is:
- much of the "research" that is done on Shuttle flights could be done just as well by unmanned missions; and
- "reusable spacecraft" is an oxymoron at the current state of technology (even ignoring pork boondoggles like Morton Thiokol in Utah)
.
Time to ax the Shuttle program. Give NASA some real money. Move the little experiments to the various LEO launches on small vehicles. Use heavy lift rockets like Energiya and Ariane while NASA designs and contracts out a US design, perhaps an updated SaturnV or something. To hell with jingoistic crap like "giving up the space race to the Russians and Europeans" -- let's not cut off our noses to spite our faces.And let's not forget that space travel for humans is still very much an experimental thing. "There be dragons -- expect to die!" There still will be no dearth of volunteers for astronaut positions.
Unlimited growth == Cancer.
God - do we have to have this argument AGAIN on slashdot?
The problem with private enterprise is that it expects rewards from its funding - rewards that generate $$$, not scientific knowledge or nationalistic pride, but cold hard cash. The problem with space is that there is as-of-yet, no viable way to make $$$ out there. Tourism is the only industry that's already made a start in space, but its first steps were shaky, it relied on a publically-funded infrastructure, and it has yet to progress any further. As for mining, there is nothing up there that we can't get down here for cheaper. Some might point to the He-3 resources on the moon, but these are not needed at all except for in undeveloped nuclear fusion technology.
I suppose there is one commercial industry that has been succesful in space: the sattelite communication/telecommunications industry. However, private interests are not going to progress beyond the sorts of sattelites we are currently flying, let alone go anywhere near manned flight on there own.
In conclusion, I would argue that private interest is not an "easier beast to summon." In fact, I would say that it is much more difficult to raise funds for space exploration through private means than it is to get public support. A space race with China would generate the neccesary support very quickly, and we might start seeing some projects come to fruition rather than being nearly completed only to be scrapped for going over-budget, and then being restarted a few years later to satiate the military-industrial complex. The only alternative that I can see having any success in space other than publically-funded programs is philanthropy. If some very rich people got together and started offering more prizes similar to the X-prize, we could see some actual development. It worked in aviation, the only thing holding it back for space is that the prizes need to be that much bigger to make it worthwhile.
In short, space exploration's only hope lies in publically funded programs or philanthropic rewards, not in the commercial exploitation of resources that don't exist.
The data is lost huh? Then build a Saturn VI!
here's my case:
1. Cryogenic turbo pump design and reliability has improved significantly since the early 70's
2. all the data we need is just lying around in space museums and outdoor rocket gardens. i think i saw something on the net that had an SV laying on it's side. not to mention recoverd apollo capsuls.
3 Materials technology both metalurgical and especially composite is well in advance of what they had available in the 60's. All we really neeed is the dimensions of this stuff
4. our sensors and digital control devices are much more accurate and faster reacting and can process more I/O.
5. the stages could be made reusable due to advances in materials technology giving us higher strength and lighter weight. with our miniscule electronics we could also have "smart" stages that could recover themselves to pre determined points on the globe.
6. the payload could increased because of he abovementioned wieght savings and improvements in the turbopump/engine design.
Thus we would have a Saturn VI instead of a Saturn V.