Elon Musk: Faulty Strut May Have Led To Falcon 9 Launch Failure

garyisabusyguy writes: This Forbes article provides the best analysis of the loss of the last Falcon 9 mission based on information released by Elon Musk to reporters. Highlights include:

• 1. Sound triangulation led them to identify a strut holding helium tank as root cause where the falling helium tank pinched a line causing overpressure in the LOX tank.
• 2. The failure occurred at 2,000 pounds of force, and the struts were rated at 10,000 pounds of force. They initially dismissed this as a cause until sounds triangulation pointed back to the strut
• 3. Further testing of struts in stock found one that failed at 2,000 pounds of force, with further analysis identifying poor grain structure in the metal, which caused weakness
• 4. It will be months before the next launch while SpaceX goes over procurement and QA processes all struts and bolts, and re-assesses any "near misses" with Air Force and NASA
• 5. Next launch will include failure mode software, which will allow recovery of the Dragon module during loss of the launch vehicle since they determined that it could have saved the Dragon module in this lost mission

.

Comments

Interesting way to sabotage SpaceX

[DickBreath]

Sell them faulty metal.

Or faulty parts made of metal.

It's just a thought, but would a competitor stoop to that? Even if not now, at some point in the future?

Re:Interesting way to sabotage SpaceX

[bughunter]

I believe they would. I worked for American Rocket Company in 1989. They developed hybrid engine IP that SpaceDev now owns.

Anyhow, in the late 80's the owners of what would become AmRoc decided to go into the launch business and build a single stage suborbital proof of concept vehicle. Oxidizer was helium pressurized LOX tank and steered by the injected fuel vector control (60% Hydrogen Peroxide, just a few gallons). It needed a guidance system. I was a new grad hired to help obtain it, and do some of the wiring and testing.

The vendor was in Boulder CO. I forget their name. The name no longer exists because AmRoc's chief competitor, Orbital Sciences Corporation sabotaged AmRoc's avionics contract. How? On the eve of our CDR with this Boulder company, all of the employees quit, except for the two owners and a single tech who'd started the company with them. Why did they quit? Orbital Sciences Corp (now Orbital ATK, after they acquired Alliance earlier this year) hired every one of them out from under their feet promising to open a Boulder office.

And it worked. Instead of a custom designed flight computer, gyros, telemetry, data acquisition and etc., we got an avionics system made from secondhand Japanese gyros, engineering model electronics left at the Boulder shop, and the rest from the Omega catalog. Furthermore, only two gyros were available, and in that case, the Z-axis is the only choice to go uninstrumented. Which meant that the flight profile would have to rely on a simple timer schedule: when to start steering away from vertical, when to separate payload, etc. And when to pull the umbilical cord.

All of these things contributed to the failure mode: which was that the LOX valve (a 2.5" gate valve) became encased in ice after frost from the previous day's rehearsal/test pooled around the valve and then refroze when the LOX was filled on launch day. It actually opened about 10%... enough to light the engine. And the person in charge of manually pulling the umbilicals (a payload customer, not an employee) jumped the gun and didn't wait for visual verification of liftoff... so no command could be given to close the valve and turn off the engine. As a result, the rocket sat on the pad and idled, the timer ran up to the moment when the thrust vector Peroxide started flowing... and the X and Y accelerometers saw no response... so more peroxide flowed. Until it pooled in the flame bucket and caught fire.

We sent a nice big black cloud over Santa Maria that day. You can read about it here. But we did prove how safe a hybrid is: if it had been a solid engine or liquid fueled rocket, there would have been a very large explosion. Instead we effectively had a tire fire.

Amroc laid off 90% of its employees within 2 months. Closed its doors and sold its IP to Westinghouse a couple of years after that. Westinghouse later sold it to SpaceDev. And some very happy very well paid engineers in Boulder Colorado earned some very bad karma.

Do I think that Orbital ATK would pay a third party vendor to skimp a little on Acceptance Testing of critical structural components made for SpaceX?

Why yes. Yes I do.

Re:Holy Jebus

[lgw]

Now THAT is how you summarize.

Well, missing from TFS was what SpaceX is doing to prevent this specific problem in the future:

To avoid this type of accident occurring in the future, the company will now individually test every strut it installs on a Falcon 9, regardless of its material specification. It's also considering a different material for its bolts, as the bolthole was the likely site of failure, and will likely switch strut suppliers.

The first thing any engineer (in any discipline) needs to learn when starting a real job is "the vendor is a lying bastard". I think it will work out substantially cheaper in the long run to test every strut rather than to go crazy with the material specification. Accept the universal truth that the vendor is a lying bastard, test as needed, and get on with life. If SpaceX ever reaches their reusability goal, the cost of all the testing will be spread across many flights anyway.

Re:Transparency

[Anonymous Coward]

Conveniently, this part has about a 5x margin (it was implied elsewhere that the 2000lbf is also right about the design load). They could test to two or three times the design load and not even be close to the rating.

Re:Holy Jebus

[Rei]

Also, maybe it's just because I've never worked in that industry before, maybe it's common practice in rocketry, but is anyone else impressed with the use of sound triangulation to figure out which part broke? I've never heard of that being done before.

Sad that the Falcon Heavy won't be launched until next spring, I've been really looking forward to that. Oh well...

Re:Holy Jebus

[Shinobi]

Such techniques are used in live environments today, such as factories, oil platforms etc. In fact, I'd be surprised if SpaceX DIDN'T have at least 6+ microphones or other vibration sensors relaying telemetry, and baselines from previous launches to compare with. With that baseline, you can tell if it happens to be a turbopump that malfunctions. Hell, if a strut or something would break in my brothers boat, we hear it immediately, because the overall vibration and thus overall boat noise will be altered.

Also, they clearly have decent bandwidth to the rocket during launch, given that they can have video feeds etc, and you can easily get multiple audio telemetry feeds to take less bandwidth than even a low-res video feed.

Re:Holy Jebus

[bughunter]

"the vendor is a lying bastard"

As a former aerospace systems engineer (now RF systems engineer), I found that that is certainly how the US Government and their Prime Contractors treat their suppliers. Every process in place is there to make sure you're not lying about something, cutting a corner, or inflating expenses. And the depressing thing is that every one of them is there to prevent recurrence of a dishonesty that actually took place in the past.

So even if you are one of those vendors that acts in good faith, and believes that a quality product is the best advertising, and that if your project is paid for by tax money then you're ethically obligated to do the best job possible, you get treated exactly like the asshole who made the decision to make a few extra bucks profit by not properly verifying the workmanship on a major structural component for a vehicle that he knew would eventually be manned.

And as an engineer, I know that most engineers want to act in good faith. Some are inept or inexperienced but they still have good faith. The problem lies in management. Once you get the lawyers and bean counters involved is when asshole decisions like that get made.

Re:Holy Jebus

[techno-vampire]

I don't know if the railroads still do this, but you used to see men walking down the length of a train tapping each wheel with a hammer and listening to the way they rang. They did this because if a wheel was going bad (i.e., cracking) they wouldn't sound right. It might have looked like busy-work, or featherbedding, but it prevented many train wrecks and saved countless lives.