Elon Musk: Faulty Strut May Have Led To Falcon 9 Launch Failure

garyisabusyguy writes: This Forbes article provides the best analysis of the loss of the last Falcon 9 mission based on information released by Elon Musk to reporters. Highlights include:

• 1. Sound triangulation led them to identify a strut holding helium tank as root cause where the falling helium tank pinched a line causing overpressure in the LOX tank.
• 2. The failure occurred at 2,000 pounds of force, and the struts were rated at 10,000 pounds of force. They initially dismissed this as a cause until sounds triangulation pointed back to the strut
• 3. Further testing of struts in stock found one that failed at 2,000 pounds of force, with further analysis identifying poor grain structure in the metal, which caused weakness
• 4. It will be months before the next launch while SpaceX goes over procurement and QA processes all struts and bolts, and re-assesses any "near misses" with Air Force and NASA
• 5. Next launch will include failure mode software, which will allow recovery of the Dragon module during loss of the launch vehicle since they determined that it could have saved the Dragon module in this lost mission

.

Comments

Holy Jebus

[Anonymous Coward]

Now THAT is how you summarize.

Re:Holy Jebus

[lgw]

Now THAT is how you summarize.

Well, missing from TFS was what SpaceX is doing to prevent this specific problem in the future:

To avoid this type of accident occurring in the future, the company will now individually test every strut it installs on a Falcon 9, regardless of its material specification. It's also considering a different material for its bolts, as the bolthole was the likely site of failure, and will likely switch strut suppliers.

The first thing any engineer (in any discipline) needs to learn when starting a real job is "the vendor is a lying bastard". I think it will work out substantially cheaper in the long run to test every strut rather than to go crazy with the material specification. Accept the universal truth that the vendor is a lying bastard, test as needed, and get on with life. If SpaceX ever reaches their reusability goal, the cost of all the testing will be spread across many flights anyway.

Re:Holy Jebus

[Rei]

Elon is surely really fuming about this one, as I know from past interviews with him that he really doesn't like having to source hardware from outside suppliers. He has the old "robber baron" mindset of wanting to get the whole production chain start-to-finish in house, and it's one of the things that really frustrated him when he started Tesla: at the time of the last interview I read on the subject (something like 3 or 4 years ago), he had gotten SpaceX up to 80% in-house, but Tesla was only up to 20% in-house. Car manufacture has long been all about sourcing parts from a wide range of outside suppliers.

But even at 80% in-house at SpaceX, looks like that remaining 20% still bit them : Seriously, failing at 1/5th the rated failure value? The vendor might as well have given them a cardboard cutout with the word "strut" written on it in sharpie.

Re:Holy Jebus

[Rei]

Also, maybe it's just because I've never worked in that industry before, maybe it's common practice in rocketry, but is anyone else impressed with the use of sound triangulation to figure out which part broke? I've never heard of that being done before.

Sad that the Falcon Heavy won't be launched until next spring, I've been really looking forward to that. Oh well...

Re:Holy Jebus

[Shinobi]

Using sound to probe structures, materials and devices is a pretty ancient technique, it's just advanced with the tools available. Stonemasons used to check blocks of stone to locate faults by tapping on them and listening for sounds that would indicate cracks or other faults. One of my grandfathers tried to teach me how to listen for potential rot zones in wooden walls when I was a kid also. Now, with modern tools, we can just do it much faster, and with more precision.

Re:Holy Jebus

[Shinobi]

Such techniques are used in live environments today, such as factories, oil platforms etc. In fact, I'd be surprised if SpaceX DIDN'T have at least 6+ microphones or other vibration sensors relaying telemetry, and baselines from previous launches to compare with. With that baseline, you can tell if it happens to be a turbopump that malfunctions. Hell, if a strut or something would break in my brothers boat, we hear it immediately, because the overall vibration and thus overall boat noise will be altered.

Also, they clearly have decent bandwidth to the rocket during launch, given that they can have video feeds etc, and you can easily get multiple audio telemetry feeds to take less bandwidth than even a low-res video feed.

Re:Holy Jebus

[bughunter]

"the vendor is a lying bastard"

As a former aerospace systems engineer (now RF systems engineer), I found that that is certainly how the US Government and their Prime Contractors treat their suppliers. Every process in place is there to make sure you're not lying about something, cutting a corner, or inflating expenses. And the depressing thing is that every one of them is there to prevent recurrence of a dishonesty that actually took place in the past.

So even if you are one of those vendors that acts in good faith, and believes that a quality product is the best advertising, and that if your project is paid for by tax money then you're ethically obligated to do the best job possible, you get treated exactly like the asshole who made the decision to make a few extra bucks profit by not properly verifying the workmanship on a major structural component for a vehicle that he knew would eventually be manned.

And as an engineer, I know that most engineers want to act in good faith. Some are inept or inexperienced but they still have good faith. The problem lies in management. Once you get the lawyers and bean counters involved is when asshole decisions like that get made.

Re:Holy Jebus

[Areyoukiddingme]

Nw expand that to all the parts

You mean expand it to the fraction of the 20% outsourced parts that weren't already being exhaustively tested? Most of those outsourced parts are electronic, because Elon Musk doesn't own a semiconductor factory (yet). All of those electronic parts are tested exhaustively and repeatedly, right up until the literal second of launch, as software verifies sanity in all systems. What's left are things like struts, that should be stupid-simple enough to be trustworthy. And apparently aren't.

So some fraction of a fraction of the parts will need newly exhaustive testing. Not free, but really, it's not that big of a bill.

Re:Holy Jebus

[techno-vampire]

I don't know if the railroads still do this, but you used to see men walking down the length of a train tapping each wheel with a hammer and listening to the way they rang. They did this because if a wheel was going bad (i.e., cracking) they wouldn't sound right. It might have looked like busy-work, or featherbedding, but it prevented many train wrecks and saved countless lives.

Re:Holy Jebus

[AJWM]

And the depressing thing is that every one of them is there to prevent recurrence of a dishonesty that actually took place in the past.

And sometimes it's not even dishonesty, just stupidity. This was probably more true in the earlier days of aerospace. In a book about the design and construction of the Lunar Module (I think it's Chariots for Apollo, but could be wrong) there's a section on how many of the subcontractors had to be taught clean-room and quality techniques. There's one episode where one of the Grumman managers goes out to some paint pigment company who happened to get the contract for the silver-zinc LM batteries (because they had supplies of the right materials) and sees the batteries being assembled -- in a dirty shed by people who are smoking cigarettes while doing the assembly. (They threw out the entire batch, trained everyone how to do things the aerospace way, and set up a clean room, and AFAIK there was never a problem with the LM batteries.)

On the other hand the ladies sewing space suit pressure garments at the Playtex Girdle factory knew the astronauts' lives depended on what they were doing, and did it right the first time.

That was easy

[durrr]

So after weeks of investigation it turns out it's a failure mode that even the most amateur of KSP players recognize.

When in doubt, add more struts

[Sowelu]

They should have added a lot more of them, clearly. It's not like struts have any mass.

Interesting way to sabotage SpaceX

[DickBreath]

Sell them faulty metal.

Or faulty parts made of metal.

It's just a thought, but would a competitor stoop to that? Even if not now, at some point in the future?

Re:Interesting way to sabotage SpaceX

[SuricouRaven]

Similar business practices have been used in the past. Rockefeller is a well-known example: He was an obsessive monopolist, unable to stomach the existence of any competitor to his Standard Oil empire. Among the tricks he used was to buy up manufacturers of components used in oil drilling and refining, and then refuse to sell replacements to competitors - driving them out of business when their expensive industrial machines eventually broke down and couldn't be repaired. With the competitor driven out of business Rockefeller easily purchased what was left and incorporated it into his company. He sabotaged one company by thus denying them access to oil-carrying carts for trains - and when they switched to shipping in barrels, he purchased the one company that could make a barrel sealing compound compatible with crude oil and altered the formulation to make it chemically unsuited.

Rockefeller's business practices went down in legend - you can thank him for modern antitrust laws: The first ones in the US were passed expressly in order to target him. He was the Bill Gates of the 1800s: Built up a fortune through unethical and at times outright illegal business practices, only to eventually retire and spend the rest of his life giving it away in huge grants to charitable causes.

It would be a lot harder to pull something like that today though - there are stricter regulations and laws against such things. It could be done, but it would need a great deal of legal caution and skill to avoid liability, or at least to ensure the liability lay with a collapsing shell company.

Re:Interesting way to sabotage SpaceX

[bughunter]

I believe they would. I worked for American Rocket Company in 1989. They developed hybrid engine IP that SpaceDev now owns.

Anyhow, in the late 80's the owners of what would become AmRoc decided to go into the launch business and build a single stage suborbital proof of concept vehicle. Oxidizer was helium pressurized LOX tank and steered by the injected fuel vector control (60% Hydrogen Peroxide, just a few gallons). It needed a guidance system. I was a new grad hired to help obtain it, and do some of the wiring and testing.

The vendor was in Boulder CO. I forget their name. The name no longer exists because AmRoc's chief competitor, Orbital Sciences Corporation sabotaged AmRoc's avionics contract. How? On the eve of our CDR with this Boulder company, all of the employees quit, except for the two owners and a single tech who'd started the company with them. Why did they quit? Orbital Sciences Corp (now Orbital ATK, after they acquired Alliance earlier this year) hired every one of them out from under their feet promising to open a Boulder office.

And it worked. Instead of a custom designed flight computer, gyros, telemetry, data acquisition and etc., we got an avionics system made from secondhand Japanese gyros, engineering model electronics left at the Boulder shop, and the rest from the Omega catalog. Furthermore, only two gyros were available, and in that case, the Z-axis is the only choice to go uninstrumented. Which meant that the flight profile would have to rely on a simple timer schedule: when to start steering away from vertical, when to separate payload, etc. And when to pull the umbilical cord.

All of these things contributed to the failure mode: which was that the LOX valve (a 2.5" gate valve) became encased in ice after frost from the previous day's rehearsal/test pooled around the valve and then refroze when the LOX was filled on launch day. It actually opened about 10%... enough to light the engine. And the person in charge of manually pulling the umbilicals (a payload customer, not an employee) jumped the gun and didn't wait for visual verification of liftoff... so no command could be given to close the valve and turn off the engine. As a result, the rocket sat on the pad and idled, the timer ran up to the moment when the thrust vector Peroxide started flowing... and the X and Y accelerometers saw no response... so more peroxide flowed. Until it pooled in the flame bucket and caught fire.

We sent a nice big black cloud over Santa Maria that day. You can read about it here. But we did prove how safe a hybrid is: if it had been a solid engine or liquid fueled rocket, there would have been a very large explosion. Instead we effectively had a tire fire.

Amroc laid off 90% of its employees within 2 months. Closed its doors and sold its IP to Westinghouse a couple of years after that. Westinghouse later sold it to SpaceDev. And some very happy very well paid engineers in Boulder Colorado earned some very bad karma.

Do I think that Orbital ATK would pay a third party vendor to skimp a little on Acceptance Testing of critical structural components made for SpaceX?

Why yes. Yes I do.

Transparency

[The Raven]

I am amused by the fact that a private company does better than our government at disaster transparency. That said, it is pretty stupid that Space X has not been testing random parts to confirm they meet the requested specifications. Spec verification is a basic part of outsourcing. All outsourcing fails if you can't verify that you're getting what they promised you.

Re:Transparency

[bobbied]

You can test random struts, but you can't test ALL struts or you're left with no struts. Sounds like they didn't test the right ones is all.

Quite true, but there are OTHER ways of figuring out the strength of the part other than testing them to failure or causing damage to the parts. Some of these methods are quite expensive, but effective. How you get a one part that's 1/4th the strength the design requires through a manufacturing process, onto a flight ready hardware assembly and not know it says serious problems lurk within. Seems Space-X has some defects in it's quality assurance processes, and that should scare you more than this one launch failure. They are playing way too fast and loose with quality and I'd be very worried about their efforts towards getting "human" rated are not going well.

This was a structural failure. A failure that can likely be traced to a part that was too weak for the designed loads. This isn't a DESIGN failure, it's a QUALITY failure, and that puts the whole program into question. What has to happen now is that the whole QA process needs to be revisited and revamped to prevent structural components from sneaking though which are not strong enough to do what they are designed to. THEN you have to go though your whole stock of parts, sub assemblies, and flight ready assemblies and figure out what you can verify as trustworthy using your NEW QA process, throw out the rest and order, assemble and test replacements.

This isn't an easy or quick fix...

Then there is the whole, our supplier lied to us, approach, which will be quicker to deal with, but only because you just have to obtain a batch of replacement struts, fully tested and verified, replace all existing hardware that used the old ones by either reworking the assemblies or building new ones that have the new parts.....

I'm guessing Space-X will opt for the latter in public, but unless they sue the supplier for damages, the problem really is the former, which scares me..

Re:Transparency

[Anonymous Coward]

Conveniently, this part has about a 5x margin (it was implied elsewhere that the 2000lbf is also right about the design load). They could test to two or three times the design load and not even be close to the rating.

Re:$10,000 toilet seats

[Moof123]

Yep. Once you have thousands of parts all designed to trade off as much strength for weight reduction as possible it doesn't take much of a manufacturing hiccup to cause an expensive "excursion". Vendors end up having to rigorously test every widget, and custom design it just for you.

Before long $10k each for a batch of a half dozen toilets seat that are space rated to not outgas funny chemicals that foul optics, handles 10g's, has 6 sigma of de-rating for the bolt hole strength, weighs under 500g, and is non-flammable starts sounding like a deal.

Would have saved itself

[cbhacking]

Pilot wouldn't have needed to. Dragon 2 has automatic abort capabilities (even when unmanned). It would have separated from the second stage - probably firing its SuperDraco thrusters - and then automatically deployed parachutes once it was a safe distance away.

Dragon 1 doesn't have the SuperDracos (only the much smaller Draco attitude control thrusters) so it wouldn't have been able to put as much distance between itself and the booster, but from the video and the telemetry it looks like the capsule survived the (accidental) separation anyhow. It could have deployed its parachutes and probably survived the landing, but it wasn't programmed to do so. They have added it to the Dragon 1 programming now though.

Failures that occur high enough to land under parachutes, slow enough to get away from the inevitable explosion without heavy rockets, and early enough in flight that there's no time to manually enable the landing sequence are... really, really rare in rocketry. Usually you either fail at liftoff (see Orbital's last attempt to launch Antares), fail rapidly and catastrophically during liftoff (any number of examples), or fail once in orbit (often, though not always, at stage separation). In orbit you have time to make a decision and send orders. On the launchpad you can't land safely (without abort rockets). In midair you *usually* can't get away in time (without abort rockets). This was an exception to the "in midair" usual failure case; there were nine seconds from beginning of the failure to loss of vehicle, and in fact the capsule had already tumbled free (and probably *could* have used its ACS thrusters to put some extra distance between itself and the booster.

One thought, though: what about, in the case of a pre-separation second-stage failure, executing MECO 1 (Main Engine Cut Off, when the Falcon 9 first stage kills its rockets) early and doing an emergency stage separation? Normally there's no point - the first stage on most launch vehicles has no purpose if the launch fails and nowhere to go even if it separates safely - but the Falcon 9 first stage is designed for reusability. Emergency MECO, separate the stages, use the ACS and/or grid fins to steer clear of the second stage, and then fire up the main engines again and aim for the droneship or other landing pad. You'd need to be quick about it, and it might still not work, but if it does you've saved a booster worth $70,000,000 USD. Well, that and demonstrated the first successful first stage recovery ever, but assuming that becomes as routine as Musk wants it to be...

Actually, it would have been super cool if the first successful recovery of the first stage had been an emergency abort!

Re:certified materials

[Rei]

You think having the part designed to handle five times the load it actually experienced to not be "with sufficient margin"? How much of a margin do you want them to put, 100x?

RTFA. They were doing statistical-sampling quality control testing of struts. The problem was that most of them were just fine, but there were a very small number which were totally defective and broke at a tiny fraction of their rated value. And no, SpaceX did not make the parts, it was an outside supplier. And yes, SpaceX A) will now be testing 100% of them, and B) is ditching the supplier.