Slashdot Mirror
Why Hubble Broke and How It Was Fixed
angry tapir writes "I recently had the opportunity to sit down with Charles (Charlie) Pellerin, who was NASA's director of astrophysics when the Hubble Space Telescope launched with its seemingly fatally flawed optical system. Pellerin went on to head up the servicing mission that finally fixed the telescope and for that was awarded NASA's highest honor, a Distinguished Service Medal. Since Hubble he has done a lot of thinking about the problems that led up to the error and how organizations can best avoid making similar mistakes."
The real hero of that project was a man called Story Musgrave. http://en.wikipedia.org/wiki/Story_Musgrave There was a lot of planning put into fixing it, but without him actually up there in space improvising when stuff went south, the Hubble would be useless today.
I worked at Ball Aerospace years ago and found out the real story. NASA cut the budget for Hubble so that a final optical train alignment task was never done. The engineers had designed a laser test to check the optical path but NASA wanted to save the $50000 the test would take. So until it was turned on, in space, they had no clue how bad it was. Working with NASA was tough mostly due to their arrogance.
The article mentions that the contractor was afraid to bring up problems.
That, plus the mentality from management that people who bring up problems are "troublemakers," "negative," "not team players," etc. (because they've put too much of their ego or political capital into a project) has got to be responsible for more disasters, large and small, than any other deadly combination.
I worked for a large nonprofit that blew money on doomed projects as though money grew on trees. Each time, it started with somebody, usually a contractor or somebody else who stood to gain from it, flattering the leadership that this was huge and visionary and would make or save them millions. Then the organizational mind control started, where everybody was saying that it was the greatest thing ever. Then the flawed project management started. Then when the cracks were obvious, people who pointed them out were vilified as naysayers. It was only the lower-downs who said anything because to rise, one had to be a "team player," and the organization was hierarchical enough that lower-downs were ignored. Then denial that there were problems, together with tossing more money at it (including adding more people to a software project at the last minute because that always works). Then even when the leadership [sic] team [sic] all realized there were problems, they all waited until the person responsible for the project was willing to concede defeat. because in a political environment, nobody wants to confront somebody who might retaliate
Those elements are the inevitable recipe for disaster for any project, but it's fear that drives virtually all of them. Fear of not looking good (note that the Congresscritter didn't yell about wasting taxpayer money, she yelled about being made to look bad), loss aversion, fear of admitting a mistake, fear of speaking up.
Pellerin was brave enough to do something technically illegal and scrape up the funds for servicing it.
That is what a leader does.
I wanted to joke about the PhDs getting drunk at their desks, but there are a couple of gems in the text:
We see this very clearly when discussing evoting.
Then towards the end there is an interesting analogy of the Shuttle accidents with a Korean airline company having an extreme crash rate, referring to people put under too much pressure, and irrational .
It's actually a good read for people interested in managing.
NB: The message above might reflect my opinion right now, but not necessarily tomorrow or next year.
Mars Impactor was designed to hit the Red Planet at 600 mph, but because of The French, it hit the planet at 1000 kph instead, ruining the mission.
I don't remember the last time I so thoroughly enjoyed and appreciated an article. To the original poster, and the /. editors that approved this - thanks. This one was a winner.
The basis of the units are irrelevant; consistency in their use is. Unless you're able to tell me that the length of a path travelled by light in a vacuum in 1/299,792,458 is directly related to landing a probe on Mars.
Gamingmuseum.com: Give your 3D accelerator a rest.
It was because people weren't paying attention. Had their been a second set of eyes regarding the specs for the control system, and then a review to make sure the programming was correct (it was the control system that had inaccurate data), there would have been no issue. There is the misplaced notion that somehow S.I. is more accurate than Imperial. It isn't. You could create anything to be the basis for a standard of measurement. As long as your measurements (and methods thereof) are reproducible, and your instruments of measurement are calibrated to a traceable standard, all is good. While it would be convenient for the US to adopt S.I. wholly, it doesn't make measurements in Imperial units any less accurate. It does make instrument manufacturers richer, though, as companies working with both systems have to buy two sets of instruments, two sets of standards, etc.
You should read the article, because your comment is exactly the kind of thing he is talking about. Technical people who think they have found a technical problem, therefore the solution is to correct that problem. If the problem was really that measurement system the US uses is 'wrong', then how can the US have so many successful space missions? The problem is not that there are multiple measurement systems, or that one is somehow superior to the other. The problem is that the teams did not communicate successfully - not a technical problem at all. And don't say 'well, if the stupid US would use the same system as the rest of the world it wouldn't be a problem', because that just shows you completely missed the point. The point is that there was ineffective communication - a leadership problem - not simply a technical problem.
I attended an astronomy conference a year ago that included a presentation from a NASA guy on the mars rovers. He had a few disparaging things to say about Lockheed-Martin, including blaming them for the Mars Climate Orbiter failure. He said their contract included a statement to recalibrate the thruster in the metric system but they failed to do so. (Of course, he neglected to mention that NASA was managing the project and failed to catch the error.) He also said one of the rovers drove by the heat shield (built by Lockheed-Martin) from the rover landing and there was a big disagreement over examining the heat shield up close to see how well it held up. Lockheed-Martin wanted the data but wanted to keep it secret on the grounds it was a proprietary design. NASA said all their data is public so it's either we drive by without looking, or we take a look and release all the data. They eventually did the latter.
One more thing -- the same conference included a presentation by a professional astronomer who had overseen the building of an observatory in Chile. He had disparaging things to say about NASA -- that their cost estimate was 10X over what he eventually spent on the project. Guess it all depends on your point of view.
That's exactly my point. NASA *does* use SI units... just not consistently. It has suppliers that use non-SI units. Those units are also self-inconsistent in some cases (for example, the size of a gallon).
NASA specified SI.
Supplier did not supply SI, since it bases its measurements on US system.
Problems.
Yes, it was a communication and management error, but not entirely. It has been standard in scientific settings to use SI units for years and years. Failure to use them *especially when specifically outlined by the design brief* is not just a "communications problem" - it's a fundamental error in the product that was delivered unfit for purpose.
Well, you did miss the point. It was ENTIRELY a communications problem. Saying 'NASA specified SI' does not mean there was effective communication. That's like saying just because a teacher gave a lecture all the students have learned the information. How do you know the information was received and understood? How do you know that information was received and understood by every single person working on the project?
It is obvious that somewhere along the line there was a breakdown in communication. It could have been between NASA and the supplier, it could have been internal to the supplier, but somewhere communication failed, and that failure is what lead directly to the disaster. And the poorest excuse for bad communication is 'you should have known'.
Note that this failure could have just as easily occurred even if SI was being used. What if one party thought they were getting a rate specified in cm/sec, but the other party thought it was supposed to be in mm/sec? All SI units, just as spectacular a failure. Didn't an Arianne rocket blow up because of something very similar to this?
I think the article was in some ways flawed. It gave a good description of how the error occurred. Then it moved on to a huge tirade against the focus on "individual abilities" which it blames for the whole error. Firstly, even taking the description of how the error occurred at face value, it is not at all clear that the error had anything to do with a focus on "individual abilities". On the contrary, it seems this was just an instance of really poor management that - due to cost overruns - pushed their employees to work harder, to the point that they lost their focus on quality and maybe even started cutting corners in the fabrication process. This has absolutely nothing to do with a focus on "individual abilities". However, let me address the "anti-individual abilities agenda" anyway.
The anti-individual abilities agenda is routinely promoted by managers, project managers and other people engaged in the management layers (management consultants, business schools etc.). The motive is pretty clear: Many bosses don't like admitting that the success of their project comes down to individual abilities of a few core members on the project. After all, what is the value of management then, they ask? It's like the tail wagging the dog.
However, this is just denying reality. I can firmly say that on any project of major size I worked on, the was a few 5-10% of people on the project running the show. This in itself is not very surprising, what is surprising is the fact that these 5-10% were not centered at the top of the pyramid. Rather, it was evenly spread out over all 'layers' from 'highest to lowest'. These people (by virtue of their skills and dedication to the project, something that is often lacking with the project management itself!) automatically assume a role of authorities whether management likes it or not. It's simply the only way to get things done. Let's face it, on any project there's going to be a lot of 9-5'ers that don't really care. They are never the ones driving the car, nor should they. It's the 5-10% who has both the ability to and the interest in getting the job done that counts. Those that dream about the project at night and who feel their personal honour is at stake in making it succeed. Also, as Fred Brooks noted in 'a mythical man month', some (sub)projects are like surgery. You need one highly skilled person to be in charge and carry out the job, and the rest of the team members are really just accessories of that person. Their contribution can be important of course, but at the end of the day, all choices, responsibility resides with the 'surgeon' etc.
I think the lesson to be learned from these observations is that management needs to accept that this is the structure that projects will generally fall into, no matter what they do. The job of management is to get the best result out of it. On projects with poor management that creates obstacles for progress and makes lots of bad choices (this often happens on politically infested projects as well as on projects where management doesn't have a clue about the technical aspects), often the project finds a way to completely bypass management. Decisions by management may be outright ignored, or important decisions are never brought up to this level but are just made behind the scenes. This is a very dangerous situation since important decisions may not be properly reviewed and may not even be known by all stake-holders. While most decisions taken may have been correct, it takes just one bad decision to jeopardize the project, and problems related to this kind of "skunkworks decisions" tend to surface very late where they may cause huge problems, sometimes disasters.
The job of management is to embrace the individual abilities, and to listen carefully (but of course not uncritically) to arguments brought forward, no matter if it is from a project manager or a "lowly" techie. They need to make a decent effort to try to understand what they are talking about, even if the explanations are not always clear and even if it can sometimes be highly technical.
Contrary to popular belief, the mixup was not an SI vs English units problem. The problem was that the numbers were passed from Lockheed to NASA without units. Without the actual units jotted down after the numbers, the Lockheed people knew the units were lb-f. The NASA people assumed the units were Newtons.
It's an important distinction because the same error can happen even if you work entirely in SI units. If I write down a number in kilonewtons but fail to write down the units, and you assume it is just newtons, we end up with the same problem. I've seen this happen countless times in the lab and while tutoring, with kids plugging grams into an equation when they're supposed to be using kg. (Which BTW is one stupid thing about SI units - really confuses the kids that the base unit for everything else has no prefix, but the base unit for mass is a kilo-gram.) Fortunately, forcing them to write down the units after every number usually takes care of this problem.
In science and engineering, any time you see a number without units, your immediate reaction should be to ask the person who provided the numbers what the units are. (Actually you should be ripping him a new one for failing to write down the units, dimensionless numbers excepted.) Never assume the units, always ask.
But that's the point - they are a standard (and ever since their inception, the standards body has been looking for ways to define them based on invariable things, rather than on arbitrary things like a mass of platinum/iridium alloy, or a metal rod that is a certain length.
SI as a system for standardising units across science is not controversial, or even new - the fact that a multi million dollar space mission can fail so spectacularly because one supplier was using imperial units is just unforgivably moronic. They can blame it on "communication errors" all they want - the fact is that the people who built the thing didn't even *look* at the units on the design brief - they just assumed everything was in imperial units.
The REAL real hero would be the advisor against naming your child Story.
"Love heals scars love left." -- Henry Rollins