Slashdot Mirror


Elon Musk: Faulty Strut May Have Led To Falcon 9 Launch Failure

garyisabusyguy writes: This Forbes article provides the best analysis of the loss of the last Falcon 9 mission based on information released by Elon Musk to reporters. Highlights include:
  • 1. Sound triangulation led them to identify a strut holding helium tank as root cause where the falling helium tank pinched a line causing overpressure in the LOX tank.
  • 2. The failure occurred at 2,000 pounds of force, and the struts were rated at 10,000 pounds of force. They initially dismissed this as a cause until sounds triangulation pointed back to the strut
  • 3. Further testing of struts in stock found one that failed at 2,000 pounds of force, with further analysis identifying poor grain structure in the metal, which caused weakness
  • 4. It will be months before the next launch while SpaceX goes over procurement and QA processes all struts and bolts, and re-assesses any "near misses" with Air Force and NASA
  • 5. Next launch will include failure mode software, which will allow recovery of the Dragon module during loss of the launch vehicle since they determined that it could have saved the Dragon module in this lost mission

.

46 of 220 comments (clear)

  1. Holy Jebus by Anonymous Coward · · Score: 5, Funny

    Now THAT is how you summarize.

    1. Re:Holy Jebus by lgw · · Score: 4, Interesting

      Now THAT is how you summarize.

      Well, missing from TFS was what SpaceX is doing to prevent this specific problem in the future:

      To avoid this type of accident occurring in the future, the company will now individually test every strut it installs on a Falcon 9, regardless of its material specification. It's also considering a different material for its bolts, as the bolthole was the likely site of failure, and will likely switch strut suppliers.

      The first thing any engineer (in any discipline) needs to learn when starting a real job is "the vendor is a lying bastard". I think it will work out substantially cheaper in the long run to test every strut rather than to go crazy with the material specification. Accept the universal truth that the vendor is a lying bastard, test as needed, and get on with life. If SpaceX ever reaches their reusability goal, the cost of all the testing will be spread across many flights anyway.

      --
      Socialism: a lie told by totalitarians and believed by fools.
    2. Re:Holy Jebus by garyisabusyguy · · Score: 2

      Yeah, I tried to cover that under item 4 with "goes over procurement and QA processes all struts and bolts"

      Admittedly not a "smoking gun" sort of statement, just *ahem* a summary

      --
      Wherever You Go, There You Are
    3. Re:Holy Jebus by Ichijo · · Score: 2

      3. Further testing of struts in stock found one that failed at 2,000 pounds of force

      I think it will work out substantially cheaper in the long run to test every strut rather than to go crazy with the material specification.

      That reminds me of the joke about the boy scout who would test all his matches and save the ones that lit successfully.

      --
      Any sufficiently unpopular but cohesive argument is indistinguishable from trolling.
    4. Re:Holy Jebus by Rei · · Score: 4, Insightful

      Elon is surely really fuming about this one, as I know from past interviews with him that he really doesn't like having to source hardware from outside suppliers. He has the old "robber baron" mindset of wanting to get the whole production chain start-to-finish in house, and it's one of the things that really frustrated him when he started Tesla: at the time of the last interview I read on the subject (something like 3 or 4 years ago), he had gotten SpaceX up to 80% in-house, but Tesla was only up to 20% in-house. Car manufacture has long been all about sourcing parts from a wide range of outside suppliers.

      But even at 80% in-house at SpaceX, looks like that remaining 20% still bit them : Seriously, failing at 1/5th the rated failure value? The vendor might as well have given them a cardboard cutout with the word "strut" written on it in sharpie.

      --
      "You see, Government is a system that is based on weapons." -- Timster
    5. Re:Holy Jebus by Rei · · Score: 5, Interesting

      Also, maybe it's just because I've never worked in that industry before, maybe it's common practice in rocketry, but is anyone else impressed with the use of sound triangulation to figure out which part broke? I've never heard of that being done before.

      Sad that the Falcon Heavy won't be launched until next spring, I've been really looking forward to that. Oh well...

      --
      "You see, Government is a system that is based on weapons." -- Timster
    6. Re:Holy Jebus by camperdave · · Score: 2

      Also, maybe it's just because I've never worked in that industry before, maybe it's common practice in rocketry, but is anyone else impressed with the use of sound triangulation to figure out which part broke? I've never heard of that being done before.

      *camperdave raises his hand.
      I am impressed that they were able to do this. I imagine that this would be difficult under the best of circumstances, but inside a rocket during a launch? That's not the type of environment I would expect to hear anything apart from the roar of the engines. I'm picturing 3D renderings showing simulated sound waves, experts trying to line up similar sonic wave-fronts to compute timings, and lots of computer time.

      --
      When our name is on the back of your car, we're behind you all the way!
    7. Re:Holy Jebus by Shinobi · · Score: 4, Informative

      Using sound to probe structures, materials and devices is a pretty ancient technique, it's just advanced with the tools available. Stonemasons used to check blocks of stone to locate faults by tapping on them and listening for sounds that would indicate cracks or other faults. One of my grandfathers tried to teach me how to listen for potential rot zones in wooden walls when I was a kid also. Now, with modern tools, we can just do it much faster, and with more precision.

    8. Re:Holy Jebus by Ol+Olsoc · · Score: 2

      The first thing any engineer (in any discipline) needs to learn when starting a real job is "the vendor is a lying bastard". I think it will work out substantially cheaper in the long run to test every strut rather than to go crazy with the material specification.

      Spacex, welcome to NASA's world.

      I'm rooting for them for sure, but After learning the hard lessons, and the inevitable expenses to correct them, their costs will increase. dramatically.

      --
      The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
    9. Re:Holy Jebus by Ol+Olsoc · · Score: 2

      Ah yes, the Gregory House/Fox Mulder method: everybody lies, trust no one.

      That's certainly a lot safer than the alternative.

      --
      The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
    10. Re:Holy Jebus by Shinobi · · Score: 3, Interesting

      Such techniques are used in live environments today, such as factories, oil platforms etc. In fact, I'd be surprised if SpaceX DIDN'T have at least 6+ microphones or other vibration sensors relaying telemetry, and baselines from previous launches to compare with. With that baseline, you can tell if it happens to be a turbopump that malfunctions. Hell, if a strut or something would break in my brothers boat, we hear it immediately, because the overall vibration and thus overall boat noise will be altered.

      Also, they clearly have decent bandwidth to the rocket during launch, given that they can have video feeds etc, and you can easily get multiple audio telemetry feeds to take less bandwidth than even a low-res video feed.

    11. Re:Holy Jebus by FatdogHaiku · · Score: 2, Funny

      Please, please tell me it was 3D printed so we can work that in somewhere...

      --
      You have the right to remain sentient. If you give up the right to remain sentient, you will be elected to public office
    12. Re:Holy Jebus by AaronW · · Score: 2

      I have a friend who helped design the Tesla drive train. He said they do so much in house since they have a lot of problems with components from 3rd parties, especially China. Look at all the transmission failures the Roadster had until they did their own in-house design.

      It also allows for them to make changes quickly. Apparently they make tweaks to the design of the car almost every week which would be impossible to do with a heavy reliance on 3rd parties for manufacturing.

      --
      This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
    13. Re:Holy Jebus by bughunter · · Score: 5, Interesting

      "the vendor is a lying bastard"

      As a former aerospace systems engineer (now RF systems engineer), I found that that is certainly how the US Government and their Prime Contractors treat their suppliers. Every process in place is there to make sure you're not lying about something, cutting a corner, or inflating expenses. And the depressing thing is that every one of them is there to prevent recurrence of a dishonesty that actually took place in the past.

      So even if you are one of those vendors that acts in good faith, and believes that a quality product is the best advertising, and that if your project is paid for by tax money then you're ethically obligated to do the best job possible, you get treated exactly like the asshole who made the decision to make a few extra bucks profit by not properly verifying the workmanship on a major structural component for a vehicle that he knew would eventually be manned.

      And as an engineer, I know that most engineers want to act in good faith. Some are inept or inexperienced but they still have good faith. The problem lies in management. Once you get the lawyers and bean counters involved is when asshole decisions like that get made.

      --
      I can see the fnords!
    14. Re:Holy Jebus by Areyoukiddingme · · Score: 4, Informative

      Nw expand that to all the parts

      You mean expand it to the fraction of the 20% outsourced parts that weren't already being exhaustively tested? Most of those outsourced parts are electronic, because Elon Musk doesn't own a semiconductor factory (yet). All of those electronic parts are tested exhaustively and repeatedly, right up until the literal second of launch, as software verifies sanity in all systems. What's left are things like struts, that should be stupid-simple enough to be trustworthy. And apparently aren't.

      So some fraction of a fraction of the parts will need newly exhaustive testing. Not free, but really, it's not that big of a bill.

    15. Re:Holy Jebus by techno-vampire · · Score: 3, Interesting

      I don't know if the railroads still do this, but you used to see men walking down the length of a train tapping each wheel with a hammer and listening to the way they rang. They did this because if a wheel was going bad (i.e., cracking) they wouldn't sound right. It might have looked like busy-work, or featherbedding, but it prevented many train wrecks and saved countless lives.

      --
      Good, inexpensive web hosting
    16. Re:Holy Jebus by AJWM · · Score: 4, Informative

      And the depressing thing is that every one of them is there to prevent recurrence of a dishonesty that actually took place in the past.

      And sometimes it's not even dishonesty, just stupidity. This was probably more true in the earlier days of aerospace. In a book about the design and construction of the Lunar Module (I think it's Chariots for Apollo, but could be wrong) there's a section on how many of the subcontractors had to be taught clean-room and quality techniques. There's one episode where one of the Grumman managers goes out to some paint pigment company who happened to get the contract for the silver-zinc LM batteries (because they had supplies of the right materials) and sees the batteries being assembled -- in a dirty shed by people who are smoking cigarettes while doing the assembly. (They threw out the entire batch, trained everyone how to do things the aerospace way, and set up a clean room, and AFAIK there was never a problem with the LM batteries.)

      On the other hand the ladies sewing space suit pressure garments at the Playtex Girdle factory knew the astronauts' lives depended on what they were doing, and did it right the first time.

      --
      -- Alastair
    17. Re: Holy Jebus by perryizgr8 · · Score: 2

      Real life is the highest fidelity 3D simulation!

      I'm stealing this for my sig.

      --
      Wealth is the gift that keeps on giving.
    18. Re:Holy Jebus by TWX · · Score: 2

      You realize that your car has at least one knock sensor on the engine, right? After market retail for an OEM replacement is usually around $100. It is able to handle the noise off of the alternator and all of the rest of the electrical and electronics plus all of the fluids, rotational mass, bearings, and anything else that is expected to make noise, can understand the difference between a combustion premature detonation and a rock bouncing off of the oil pan plus any other of a multitude of impacts noises, and probably costs the automaker just a few dollars to manufacture.

      After reflecting on it I'm not exactly surprised. Their later independent confirmation of a not-yet-flown part's failure through destructive test helps reinforce the likelihood of this. I'm glad they found something, it would be nice if we had competitive space launches to make the costs come down.

      --
      Do not look into laser with remaining eye.
    19. Re:Holy Jebus by thinkwaitfast · · Score: 2

      This is very common and what FFTs are used for. One hypothetical example would be to attach a few hundred chip sized 3-d accelerometers and send back the values of each. Here's an example, the ADXL335 Say you have 1000 of these distributed throughout the vehicle. A breaking bolt would cause a small shockwave that would easily be visible as a blip in the frequency spectrum. Each blip has a timestamp and known position, so you just triangulate back to the origin.

    20. Re:Holy Jebus by KGIII · · Score: 2

      In a koan:

      Eventually, every young engineer realizes "wait, we're a vendor too", and... the young engineer is enlightened.

      --
      "So long and thanks for all the fish."
    21. Re:Holy Jebus by halltk1983 · · Score: 2

      That's, of course, why every strut they tested failed in exactly the same way, including all those on the successful launches as well, and not that they had to test an 'enormous' number of them to find one that failed in the same way. Or perhaps it was the other way around.

      --
      Watch for Penguins, they eat Apples and throw rocks at Windows.
  2. That was easy by durrr · · Score: 5, Funny

    So after weeks of investigation it turns out it's a failure mode that even the most amateur of KSP players recognize.

  3. When in doubt, add more struts by Sowelu · · Score: 5, Funny

    They should have added a lot more of them, clearly. It's not like struts have any mass.

  4. Something is fishy by Pascoea · · Score: 2
    http://www.forbes.com/sites/al...

    the heilum bottle would have shot to the top of the tank at high speed

    That sounds a lot different than "a hose may have been pinched" Has anyone been able to find audio of the actual conversation?

    1. Re:Something is fishy by swillden · · Score: 2

      This one seems to be a pretty decent explanation: http://www.orlandosentinel.com...

      Heh. One bit of that article made me chuckle (emphasis mine):

      The explosion destroyed the $70 million rocket, its Dragon I capsule and 4,000 pounds of supplies that was headed to the International Space Station. None of it was insured.

      I can just imagine Elon going to his local insurance agent, trying to get coverage for an experimental vehicle carrying 400 tons of rocket fuel and LOX.

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
    2. Re:Something is fishy by Firethorn · · Score: 2

      Lloyd's of London will insure anything. You might just not like the premiums.

      In this case, self insurance makes sense.

      --
      I don't read AC A human right
  5. Interesting way to sabotage SpaceX by DickBreath · · Score: 4, Interesting

    Sell them faulty metal.

    Or faulty parts made of metal.

    It's just a thought, but would a competitor stoop to that? Even if not now, at some point in the future?

    --

    I'll see your senator, and I'll raise you two judges.
    1. Re:Interesting way to sabotage SpaceX by SuricouRaven · · Score: 4, Informative

      Similar business practices have been used in the past. Rockefeller is a well-known example: He was an obsessive monopolist, unable to stomach the existence of any competitor to his Standard Oil empire. Among the tricks he used was to buy up manufacturers of components used in oil drilling and refining, and then refuse to sell replacements to competitors - driving them out of business when their expensive industrial machines eventually broke down and couldn't be repaired. With the competitor driven out of business Rockefeller easily purchased what was left and incorporated it into his company. He sabotaged one company by thus denying them access to oil-carrying carts for trains - and when they switched to shipping in barrels, he purchased the one company that could make a barrel sealing compound compatible with crude oil and altered the formulation to make it chemically unsuited.

      Rockefeller's business practices went down in legend - you can thank him for modern antitrust laws: The first ones in the US were passed expressly in order to target him. He was the Bill Gates of the 1800s: Built up a fortune through unethical and at times outright illegal business practices, only to eventually retire and spend the rest of his life giving it away in huge grants to charitable causes.

      It would be a lot harder to pull something like that today though - there are stricter regulations and laws against such things. It could be done, but it would need a great deal of legal caution and skill to avoid liability, or at least to ensure the liability lay with a collapsing shell company.

    2. Re:Interesting way to sabotage SpaceX by Ol+Olsoc · · Score: 2, Funny

      Similar business practices have been used in the past. Rockefeller is a well-known example: He was an obsessive monopolist, unable to stomach the existence of any competitor to his Standard Oil empire. Among the tricks he used was to buy up manufacturers of components used in oil drilling and refining, and then refuse to sell replacements to competitors - driving them out of business when their expensive industrial machines eventually broke down and couldn't be repaired.

      Sounds like the invisible hand of the free market in action.

      --
      The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
    3. Re:Interesting way to sabotage SpaceX by bughunter · · Score: 5, Interesting

      I believe they would. I worked for American Rocket Company in 1989. They developed hybrid engine IP that SpaceDev now owns.

      Anyhow, in the late 80's the owners of what would become AmRoc decided to go into the launch business and build a single stage suborbital proof of concept vehicle. Oxidizer was helium pressurized LOX tank and steered by the injected fuel vector control (60% Hydrogen Peroxide, just a few gallons). It needed a guidance system. I was a new grad hired to help obtain it, and do some of the wiring and testing.

      The vendor was in Boulder CO. I forget their name. The name no longer exists because AmRoc's chief competitor, Orbital Sciences Corporation sabotaged AmRoc's avionics contract. How? On the eve of our CDR with this Boulder company, all of the employees quit, except for the two owners and a single tech who'd started the company with them. Why did they quit? Orbital Sciences Corp (now Orbital ATK, after they acquired Alliance earlier this year) hired every one of them out from under their feet promising to open a Boulder office.

      And it worked. Instead of a custom designed flight computer, gyros, telemetry, data acquisition and etc., we got an avionics system made from secondhand Japanese gyros, engineering model electronics left at the Boulder shop, and the rest from the Omega catalog. Furthermore, only two gyros were available, and in that case, the Z-axis is the only choice to go uninstrumented. Which meant that the flight profile would have to rely on a simple timer schedule: when to start steering away from vertical, when to separate payload, etc. And when to pull the umbilical cord.

      All of these things contributed to the failure mode: which was that the LOX valve (a 2.5" gate valve) became encased in ice after frost from the previous day's rehearsal/test pooled around the valve and then refroze when the LOX was filled on launch day. It actually opened about 10%... enough to light the engine. And the person in charge of manually pulling the umbilicals (a payload customer, not an employee) jumped the gun and didn't wait for visual verification of liftoff... so no command could be given to close the valve and turn off the engine. As a result, the rocket sat on the pad and idled, the timer ran up to the moment when the thrust vector Peroxide started flowing... and the X and Y accelerometers saw no response... so more peroxide flowed. Until it pooled in the flame bucket and caught fire.

      We sent a nice big black cloud over Santa Maria that day. You can read about it here. But we did prove how safe a hybrid is: if it had been a solid engine or liquid fueled rocket, there would have been a very large explosion. Instead we effectively had a tire fire.

      Amroc laid off 90% of its employees within 2 months. Closed its doors and sold its IP to Westinghouse a couple of years after that. Westinghouse later sold it to SpaceDev. And some very happy very well paid engineers in Boulder Colorado earned some very bad karma.

      Do I think that Orbital ATK would pay a third party vendor to skimp a little on Acceptance Testing of critical structural components made for SpaceX?

      Why yes. Yes I do.

      --
      I can see the fnords!
  6. Transparency by The+Raven · · Score: 3, Insightful

    I am amused by the fact that a private company does better than our government at disaster transparency. That said, it is pretty stupid that Space X has not been testing random parts to confirm they meet the requested specifications. Spec verification is a basic part of outsourcing. All outsourcing fails if you can't verify that you're getting what they promised you.

    --
    "I will trust Google to 'do no evil' until the founders no longer run it." Hello Alphabet.
    1. Re:Transparency by thegarbz · · Score: 2

      Random testing can only pick up systemic faults within an entire product line not random ones. You have to try and ensure the random samples are as representative of the collective as possible which can be helped by doing things such as random testing at least one from every heat number but ultimately you will never catch them all.

      Based on the summary itself:

      Further testing of struts in stock found one that failed at 2,000 pounds of force

      emphasis mine. This implies that testing other ones would have shown no problem unless it was possible to classify all the failed ones by some common batch.

    2. Re:Transparency by bobbied · · Score: 4, Insightful

      You can test random struts, but you can't test ALL struts or you're left with no struts. Sounds like they didn't test the right ones is all.

      Quite true, but there are OTHER ways of figuring out the strength of the part other than testing them to failure or causing damage to the parts. Some of these methods are quite expensive, but effective. How you get a one part that's 1/4th the strength the design requires through a manufacturing process, onto a flight ready hardware assembly and not know it says serious problems lurk within. Seems Space-X has some defects in it's quality assurance processes, and that should scare you more than this one launch failure. They are playing way too fast and loose with quality and I'd be very worried about their efforts towards getting "human" rated are not going well.

      This was a structural failure. A failure that can likely be traced to a part that was too weak for the designed loads. This isn't a DESIGN failure, it's a QUALITY failure, and that puts the whole program into question. What has to happen now is that the whole QA process needs to be revisited and revamped to prevent structural components from sneaking though which are not strong enough to do what they are designed to. THEN you have to go though your whole stock of parts, sub assemblies, and flight ready assemblies and figure out what you can verify as trustworthy using your NEW QA process, throw out the rest and order, assemble and test replacements.

      This isn't an easy or quick fix...

      Then there is the whole, our supplier lied to us, approach, which will be quicker to deal with, but only because you just have to obtain a batch of replacement struts, fully tested and verified, replace all existing hardware that used the old ones by either reworking the assemblies or building new ones that have the new parts.....

      I'm guessing Space-X will opt for the latter in public, but unless they sue the supplier for damages, the problem really is the former, which scares me..

      --
      "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
    3. Re:Transparency by Anonymous Coward · · Score: 3, Interesting

      Conveniently, this part has about a 5x margin (it was implied elsewhere that the 2000lbf is also right about the design load). They could test to two or three times the design load and not even be close to the rating.

    4. Re:Transparency by bobbied · · Score: 2, Insightful

      That said, it is pretty stupid that Space X has not been testing random parts

      So uh... "Further testing of struts in stock found one that failed at 2,000 pounds of force"

      Sounds to me like they likely were testing them at random and then decided to start testing a vastly significant number of them to troubleshoot.

      Which tells me they may have a QA process issue that compounds a design issue. NEVER do you take a single failure risk in a human rated system's structural components unless you are SURE the components are sound. So if this strut was a critical structural component without any backup and you are not totally sure it can take the load before you fly the assembly, it's a process problem.

      They do want to make this a human rated system right? That goal is in serious jeopardy now. You cannot afford to "learn on the fly" like this and arrive at a human rated design. You are going to build and launch a LOT of vehicles, fixing the problems as they crop up, before the statistics will be good enough. The cheaper way is to design sufficient safety margins into the system, do a few launches and deal with the issues as they arise, but unless you REALLY mess up, any system you hope to human rate is not going to have this many loss of vehicle failures... Structural failures should NEVER happen if you are on this track, because structure should be the easiest thing to get right because it's testable on the ground...

      IMHO Space X is doing this wrong, and a structural failure is a serious problem that goes beyond this one failure....

      --
      "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
    5. Re:Transparency by Kjella · · Score: 2, Insightful

      Statistics is your friend in this case. Random testing should show a large standard deviation (assuming they test to failure). You should then be able to calculate the probability of failure.

      No, just no. Statistics helps if you have a process variation that approximately follows a normal distribution. It does nothing to protect you against freak failure, like say you're making creme brulee using a torch to caramelize the sugar on top but the spark to light it is only 99.9% reliable. That means 999 out of 1000 desserts will be fine and one will be a total failure, but you don't know it until it happens. Failing at 1/5th the design load is clearly outside any normal variation and tells you none of these struts can be relied on no matter what their average or standard deviation is.

      --
      Live today, because you never know what tomorrow brings
    6. Re:Transparency by DerekLyons · · Score: 2

      I am amused by the fact that a private company does better than our government at disaster transparency.

      Outside of classified launches... when exactly was the government not this transparent (and more)?

  7. $10,000 toilet seats by MikeMo · · Score: 2

    THIS is why aerospace and aeronatical parts cost so dang much.

    1. Re:$10,000 toilet seats by Moof123 · · Score: 4, Insightful

      Yep. Once you have thousands of parts all designed to trade off as much strength for weight reduction as possible it doesn't take much of a manufacturing hiccup to cause an expensive "excursion". Vendors end up having to rigorously test every widget, and custom design it just for you.

      Before long $10k each for a batch of a half dozen toilets seat that are space rated to not outgas funny chemicals that foul optics, handles 10g's, has 6 sigma of de-rating for the bolt hole strength, weighs under 500g, and is non-flammable starts sounding like a deal.

  8. Would have saved itself by cbhacking · · Score: 4, Informative

    Pilot wouldn't have needed to. Dragon 2 has automatic abort capabilities (even when unmanned). It would have separated from the second stage - probably firing its SuperDraco thrusters - and then automatically deployed parachutes once it was a safe distance away.

    Dragon 1 doesn't have the SuperDracos (only the much smaller Draco attitude control thrusters) so it wouldn't have been able to put as much distance between itself and the booster, but from the video and the telemetry it looks like the capsule survived the (accidental) separation anyhow. It could have deployed its parachutes and probably survived the landing, but it wasn't programmed to do so. They have added it to the Dragon 1 programming now though.

    Failures that occur high enough to land under parachutes, slow enough to get away from the inevitable explosion without heavy rockets, and early enough in flight that there's no time to manually enable the landing sequence are... really, really rare in rocketry. Usually you either fail at liftoff (see Orbital's last attempt to launch Antares), fail rapidly and catastrophically during liftoff (any number of examples), or fail once in orbit (often, though not always, at stage separation). In orbit you have time to make a decision and send orders. On the launchpad you can't land safely (without abort rockets). In midair you *usually* can't get away in time (without abort rockets). This was an exception to the "in midair" usual failure case; there were nine seconds from beginning of the failure to loss of vehicle, and in fact the capsule had already tumbled free (and probably *could* have used its ACS thrusters to put some extra distance between itself and the booster.

    One thought, though: what about, in the case of a pre-separation second-stage failure, executing MECO 1 (Main Engine Cut Off, when the Falcon 9 first stage kills its rockets) early and doing an emergency stage separation? Normally there's no point - the first stage on most launch vehicles has no purpose if the launch fails and nowhere to go even if it separates safely - but the Falcon 9 first stage is designed for reusability. Emergency MECO, separate the stages, use the ACS and/or grid fins to steer clear of the second stage, and then fire up the main engines again and aim for the droneship or other landing pad. You'd need to be quick about it, and it might still not work, but if it does you've saved a booster worth $70,000,000 USD. Well, that and demonstrated the first successful first stage recovery ever, but assuming that becomes as routine as Musk wants it to be...

    Actually, it would have been super cool if the first successful recovery of the first stage had been an emergency abort!

    --
    There's no place I could be, since I've found Serenity...
    1. Re:Would have saved itself by AJWM · · Score: 2

      Actually, it would have been super cool if the first successful recovery of the first stage had been an emergency abort!

      Something like this actually happened on the fourth or fifth flight of the DC-X test vehicle. It had made several successful launches, hovers, and vertical landings and the test profile had it going to higher and higher altitudes and doing some interesting maneuvers. Anyway, on the nearly fatal flight, due to wind conditions in the launch area, hydrogen gas (it was LH2/LO2 fueled) collected near the base of the rocket and ignited in a mild explosion at launch, with the shockwave blowing off some of the DC-X's airshell. The rocket continued climb-out, shedding more pieces as it went, as the observers watched. Pete Conrad (the astronaut, who as part of the DC-X team was controlling the flight from the ground) calmly clicked the "abort" sequence on the control computer. The DC-X stopped its ascent, hovered to burn off fuel (to lighten the load on the landing gear) and then landed safely.

      Try that with any other rocket. (The DC-X's engines were modified RL-10s, much more deeply throttleable than Falcon's Merlin engines.)

      --
      -- Alastair
  9. Re:certified materials by Rei · · Score: 3

    You think having the part designed to handle five times the load it actually experienced to not be "with sufficient margin"? How much of a margin do you want them to put, 100x?

    RTFA. They were doing statistical-sampling quality control testing of struts. The problem was that most of them were just fine, but there were a very small number which were totally defective and broke at a tiny fraction of their rated value. And no, SpaceX did not make the parts, it was an outside supplier. And yes, SpaceX A) will now be testing 100% of them, and B) is ditching the supplier.

    --
    "You see, Government is a system that is based on weapons." -- Timster
  10. Re:certified materials by im_thatoneguy · · Score: 2

    TESTING requires destruction of this kind of thing as I read the article. They will NOT be doing 100% testing to failure of their stock of struts, except to prove to themselves how bad their supplier really was.

    Nobody said they would be testing to failure. You can test every unit to say 150% of failure. If the material is rated for 1000% of failure then 150% should be safe. If it doesn't fail at 150% once it probably won't fail at 100% 100 times. So now you're at 99 times until mean failure instead of 100.

  11. Re:Testing regime by bughunter · · Score: 2

    After the doge meme's 15 minutes of internet fame, your sentence actually much parses.

    --
    I can see the fnords!
  12. Engineering and errors by sjbe · · Score: 2

    And as an engineer, I know that most engineers want to act in good faith. Some are inept or inexperienced but they still have good faith. The problem lies in management. Once you get the lawyers and bean counters involved is when asshole decisions like that get made.

    I'm an engineer but I'm also an accountant (aka a bean counter). I also am management in my company and I'm in charge of the engineering, production and accounting among other things. You are absolutely right that most of the time the fault for most failures ultimately is due to management decisions. At the end of the day the buck stops with them and that is how it should be. HOWEVER, management ultimately relies on the expertise of engineers and the reasoned opinions of those who report to them. If management gets bad information (happens all the time) then management decisions are more likely to be bad ones. That's not to say that management can't introduce cock-ups all their own (we all know they routinely do) but bad management decisions normally don't happen in a vacuum. Most serious screw ups in a company happen because people at multiple levels in a company made a mistake. It's not just management or engineering or accounting alone. Rockets blow up when all these mistakes made by multiple people line up in just the right way. Business is a team sport and most failures in business involve more than one person.

    Engineering is kind of like playing chess. You can see the entire board and you know (or should) what is possible at any given moment because you have close to perfect information. If you are good enough mistakes are largely predictable. Management is more like playing poker. You rarely have perfect information and you have to guess based on your assessment of the probabilities and sometimes you'll be wrong even if you play the hand perfectly. The mindsets needed for success in each for each are very different and can be hard to reconcile at times.

    It has also been my experience that there are FAR more inept engineers that most people realize. Specifically too many engineers are inept at product and process documentation. It's boring and despite its importance it tends to get overlooked and engineers often spend as little time with it as they can get away with. I run a contract manufacturing company that makes wire harnesses. I can count on my fingers the number of product drawings that I've received in the last 5 years from all our customers combined that I could manufacture without having to get substantial clarification from the engineering staff at our customer. I routinely see ambiguous dimensions, incompatible parts, improper or incomplete specifications, missing part numbers, internal part numbers (useless to anyone else), incoherent diagrams, unnecessary lookup tables, obsolete or hard to get parts, and more. Most engineers I've run into are really quite terrible at documentation. I see drawings daily that were clearly not written with the expectation that anyone else might ever read them despite the fact that someone else reading it is the entire point of the document.