Slashdot Mirror


Could a Dirty Rag Take Out a $2 Billion Satellite?

An anonymous reader writes "The alleged rescue of a U.S. military communications satellite underscores some of the weaknesses in U.S. space efforts. Quoting: 'The seven-ton “AEHF-1,” part of a planned six-satellite constellation meant to support radio communication between far-flung U.S. military units, had been in orbit just one day when the problems began. The satellite started out in a highly-elliptical, temporary orbit. The plan was to use the spacecraft’s on-board engine to boost it to a permanent, geo-stationary orbit. But when the Air Force space operators at Los Angeles Air Force Base activated the engine, nothing happened. The Government Accountability Office would later blame the failure on a rag left inside a fuel line by a Lockheed worker.'"

77 of 297 comments (clear)

  1. Lockheed gonna get sued? by Anonymous Coward · · Score: 5, Insightful

    Seems like the gov't should sue lockheed for failing to deliver the working satellite as contracted.

    Hopefully that'll happen (which will probably leave that worker jobless) and we'll get some of our tax dollars back.

    Shhh... I can dream!

    1. Re:Lockheed gonna get sued? by JonahsDad · · Score: 5, Informative

      TFA states that they are seeking compensation from Lockheed. Hopefully, that'll happen without an actual suit.

    2. Re:Lockheed gonna get sued? by davester666 · · Score: 5, Funny

      Yes, the gov't will have to pay for that space-rag now. Lockheed forgot to bill them for it.

      --
      Sleep your way to a whiter smile...date a dentist!
    3. Re:Lockheed gonna get sued? by Anonymous Coward · · Score: 4, Insightful

      Seems like the gov't should sue lockheed for failing to deliver the working satellite as contracted.

      Hopefully that'll happen (which will probably leave that worker jobless) and we'll get some of our tax dollars back.

      Shhh... I can dream!

      Lockheed wouldn't piss off their biggest spender. They'll pay back in the form of a "credit" for some kind of services that have the highest margin for Lockheed. The guy who screwed up and his boss will get fired for sure, and then they will have some business analyst examine their QA process and add a little redundancy in the inspection policies. Nothing to see here folks.

    4. Re:Lockheed gonna get sued? by Ouchie · · Score: 5, Funny

      Yes, the gov't will have to pay for that space-rag now. Lockheed forgot to bill them for it.

      The bill also included the fines levied by the TSA for failing to file an export declariation on the space rag.

      --
      "Of all the things I've lost, I miss my mind the most." ~Ozzy Osborne
    5. Re:Lockheed gonna get sued? by Anonymous Coward · · Score: 5, Insightful

      Seems like the gov't should sue lockheed for failing to deliver the working satellite as contracted.

      Hopefully that'll happen (which will probably leave that worker jobless) and we'll get some of our tax dollars back.

      Shhh... I can dream!

      Lockheed wouldn't piss off their biggest spender. They'll pay back in the form of a "credit" for some kind of services that have the highest margin for Lockheed. The guy who screwed up and his boss will get fired for sure, and then they will have some business analyst examine their QA process and add a little redundancy in the inspection policies. Nothing to see here folks.

      Isn't that what should happen? I mean, when did the world suddenly decide that anytime anyone makes an honest mistake they should be crucified for it forever? If there is restitution for lost funds as well as improvements to try to prevent a repetition of the same problem, shouldn't everyone involved be satisfied? I'm fairly certain that the OP's hope that we all get some kind of tax refund is probably not going to happen, and even if it did, you'd be talking about a few dollars per person at most.

    6. Re:Lockheed gonna get sued? by z00_miak · · Score: 5, Funny

      TFA states that they are seeking compensation from Lockheed. Hopefully, that'll happen without an actual suit.

      That would be quite the space suit.

    7. Re:Lockheed gonna get sued? by pixelpusher220 · · Score: 3, Insightful

      49% don't actually pay income tax

      Because they didn't make any money above federal poverty rates....but hey, don't let details get in the way of a perfectly overused irrelevant statistic.

      the top 1% paid 40%

      When they account for 50%+ of the income, the should be paying *more* taxes, not less.

      --
      People in cars cause accidents....accidents in cars cause people :-D
    8. Re:Lockheed gonna get sued? by m3000 · · Score: 2

      Considering the top 1% control 33% of the wealth in the country, and the bottom 50% only have 2.5%, I'm pretty OK with that distribution.

    9. Re:Lockheed gonna get sued? by Anonymous Coward · · Score: 3, Informative

      The to 5% make 38%. But this is Slashdot, it's not as though you guys are going to let something like MATH get in the way of Socialism.

    10. Re:Lockheed gonna get sued? by DJ+Jones · · Score: 5, Insightful

      Actually that whole 49% figure is misleading. The element of "time" is conveniently left out. 49% of Americans at any given "time" do not pay income tax, not because they are lazy, poor, freeloading citizens but because they are either too young to earn income or retired. In actuality, over 90% of Americans pay income tax at some point in their lifetimes.

    11. Re:Lockheed gonna get sued? by Hatta · · Score: 5, Insightful

      I mean, when did the world suddenly decide that anytime anyone makes an honest mistake they should be crucified for it forever?

      Forgetting a rag is an honest mistake. Failing to plan for honest mistakes by implementing the appropriate checks into your process is negligence.

      --
      Give me Classic Slashdot or give me death!
    12. Re:Lockheed gonna get sued? by lgw · · Score: 2, Insightful

      It is relevant that 49% don't make money above the poverty line. Fix that problem

      That problem was fixed. So they moved the poverty line up, and it will keep being moved every generation, because it's a good political football.

      Look around you and see whether America seems to have a starvation problem, or an obesity problem. Real poverty, as would be recognized in most of the rest of the world, is almost absent in America.

      No matter what you do, 20% of people will be in the bottom 20%, you know. The best system to avoid too much wealth concentration is to change inheritance taxes to be income taxes, so that spreading it out when you die is greatly encouraged. On any shorter timescale I'm not sure it matters, and since wealth and income are different things, you won't otherwise solve a wealth distribution "problem" with an income tax.

      --
      Socialism: a lie told by totalitarians and believed by fools.
    13. Re:Lockheed gonna get sued? by syousef · · Score: 3, Insightful

      I mean, when did the world suddenly decide that anytime anyone makes an honest mistake they should be crucified for it forever?

      Forgetting a rag is an honest mistake. Failing to plan for honest mistakes by implementing the appropriate checks into your process is negligence.

      The engineer following the process is not necessarily the person that created the procedure. Also even if a procedure is in place double failures do occur - they are just less likely.

      I love the way so many people are willing to judge that a man should or should not be fired based on 3 minutes of reading a slashdot story. Really enhances my faith in human nature. Hope none of you ever sit on a jury. What disciplinary action if any should be faced by various staff involved is something that would require at least weeks of investigation, IF you want to go in that direction and waste the time on a witch hunt instead of just fixing the issue.

      --
      These posts express my own personal views, not those of my employer
    14. Re:Lockheed gonna get sued? by garyebickford · · Score: 2

      ... Not to mention that almost everyone in the US is in the top 1% worldwide.

      --
      It's easier to be a result of the past, but more fun to be a cause of the future! http://www.spacefinancegroup.com/
  2. Hmm...scale does not compute. by fahrbot-bot · · Score: 5, Interesting

    blame the failure on a rag left inside a fuel line

    Must be a really small rag or really big fuel line. Seriously, how would this happen? It's a freaking satellite engine, not the shuttle main.

    --
    It must have been something you assimilated. . . .
    1. Re:Hmm...scale does not compute. by localman57 · · Score: 5, Funny

      Yeah. Damn it people! This is just rocket science, not brain surgery!

    2. Re:Hmm...scale does not compute. by Richard_at_work · · Score: 5, Informative

      You can use tiny squares of cloth, impregnated with cleaning solution, to clean the inside of valves and metal lines - gets rid of metal filings which are left over from the boring process.

      Quite easy to leave one behind. Which is why there are processes in place designed to prevent such issues.

    3. Re:Hmm...scale does not compute. by 0123456 · · Score: 3, Informative

      How do they know a rag was left in the fuel line? Do they have a sensor in the fuel line that checks for the presence of rags?

      I don't know about this case, but AFAIR NASA required forms signed in triplicate saying that any tool taken into the shuttle was later removed from it. Perhaps there's similar tracking in this case and a check showed up a rag that wasn't signed out for being removed.

      It seems to be a common problem, I'm sure I remember a couple of rocket launches which were blamed on rags in the fuel lines.

    4. Re:Hmm...scale does not compute. by ClickOnThis · · Score: 4, Insightful

      Yeah. Damn it people! This is just rocket science, not brain surgery!

      Actually, compared to rocket science, brain surgery is a walk in the park.

      --
      If it weren't for deadlines, nothing would be late.
    5. Re:Hmm...scale does not compute. by Marc_Hawke · · Score: 2

      I wonder what that says for Theoretical Physics.

      http://www.youtube.com/watch?v=lVPsBmhgjTk [big bang theory]

      --
      --Welcome to the Realm of the Hawke--
    6. Re:Hmm...scale does not compute. by gnick · · Score: 2

      If it's a rag small enough to scrub the inside of a fuel line, it could easily go unnoticed on its way onto the assembly platform. Or, if one was too large, it could have been sectioned and still taken out as "one rag." But in any case, signed in or signed out, how hard is it to test whether or not the line is partially or fully plugged? Put a controlled pressure on one end and measure flow rate on the other.

      --
      He's getting rather old, but he's a good mouse.
    7. Re:Hmm...scale does not compute. by Maximum+Prophet · · Score: 4, Interesting

      You can use tiny squares of cloth, impregnated with cleaning solution, to clean the inside of valves and metal lines - gets rid of metal filings which are left over from the boring process.

      Quite easy to leave one behind. Which is why there are processes in place designed to prevent such issues.

      So, they built a tool to make sure the rag was removed. Then they built another tool to check that the first tool was removed...

      More seriously, why wouldn't groundside testing notice that there was a rag in the line?

      --
      All ideas^H^H^H^H^Hprocesses in this post are Patent Pending. (as well as the process of patenting all postings)
    8. Re:Hmm...scale does not compute. by Richard_at_work · · Score: 4, Informative

      More seriously, why wouldn't groundside testing notice that there was a rag in the line?

      Yup, why wouldn't it?

      Obviously it didn't. Multiple times. In multiple different situations - this isn't the first space mission to be ruined because of something left where it shouldn't have been.

      The obvious answer to your question might be because it didn't block anything during testing, so there was the appearance of nothing wrong. Turn on the fuel flow, after the experience of the launch, and it might have been jostled free from wherever it had chosen to hide - from there it might be a short ride to a bottle necking point such as a crimp in the line, a sharp bend, or a valve, and thus begins the blockage.

    9. Re:Hmm...scale does not compute. by treeves · · Score: 3, Insightful

      No, not paper. These things are assembled in cleanrooms, in which ordinary paper is not allowed, due to the particles/fibers it sheds. Cleanroom cloths are usually lint-free polyester cloth squares about 8 inches on a side, IME.

      --
      ...the future crusty old bastards are already drinking the Kool-Aid.
    10. Re:Hmm...scale does not compute. by SwedishChef · · Score: 5, Insightful

      More seriously, why wouldn't groundside testing notice that there was a rag in the line?

      Some of these positioning rockets are single-use. If you test one you have to build another to replace it. And then test it. And then.....

      --
      No one ever had to evacuate a city because the solar panels broke!
    11. Re:Hmm...scale does not compute. by fuzzyfuzzyfungus · · Score: 3, Interesting

      I'm a bit surprised that it wasn't caught purely as a side effect of other procedures... If I were planning on running fuel through something important that might theoretically be contaminated with lubricants/condensation/whatever, I'd strongly consider blowing $5 worth of compressed nitrogen through it until the outflow is clean...

      Catching every little thing that might gum up the fuel lines during assembly, testing, and cleaning seems like it could be a genuinely hard problem. Doing a combination pressure test/gas flush seems like it would be a cheap, simple, brute-force solution to that entire class of potential problems...

    12. Re:Hmm...scale does not compute. by Richard_at_work · · Score: 2

      I've dealt with people at the GAO, they have a lot of expertise in house on many subjects, and aren't afraid to seek advice externally to come to decisions. It's one of those branches of government that it's actually nice to have.

    13. Re:Hmm...scale does not compute. by Solandri · · Score: 2

      I don't know about this case, but AFAIR NASA required forms signed in triplicate saying that any tool taken into the shuttle was later removed from it.

      An audit of procedures after the Challenger disaster revealed that this can actually cause the problem it's attempting to solve. When you require three people to sign off that something has been inspected, one day an inspector is a bit rushed and needs to finish by 4:30 to make it to his child's school play. He figures since two other people will be inspecting the part in addition to him, he can cheat a little and just sign off on it without actually inspecting it. After all, what are the odds that all three inspectors would sign off on it without actually inspecting the part?

      Until you think, what are the odds that all three inspectors have a child in the school play that day and want to finish early to get off by 4:30? Too few inspectors and you're vulnerable to mistakes. Too many inspectors and you're vulnerable to complacency.

    14. Re:Hmm...scale does not compute. by pipingguy · · Score: 2

      Junk left in piping can cause all kinds of problems (like, duh). That's why there are procedures to clean out the lines prior to startup. If the rag was left in oxygen or oxygen-rich piping, well...kaboom.

      One space industry insider, who spoke with The Diplomat on condition of anonymity, says lapses like the forgotten rag indicate a lack of experience in the lower ranks of U.S. space contractors. âoeIt was probably a mix all too common in the USAF programs: 80-year-old PhDs and 20-year-old college grads.â Periodic collapses in U.S. space funding, such as occurred in the 1990s, have resulted in entire missing generations of space engineers. Today, there are precious few mid-level engineers to bridge the gap between the veterans and the new hires. As a result, âoethe zero-practice grads make simple mistakes,â the insider says.

      I currently work as a design checker (but not in rocket science) - who checks the checker?

  3. The answer appears to be a yes. by jandrese · · Score: 5, Interesting

    Isn't this sort of like asking if a $5 wrench could wreck a car engine if it were left inside of a cylinder? Is anybody going to say "no"?

    And yes, I went with the car analogy right from the start. Deal with it.

    --

    I read the internet for the articles.
    1. Re:The answer appears to be a yes. by Suki+I · · Score: 3, Interesting

      Exactly! Any old blockage could prevent fuel from getting through the fuel line. Same with the oxidizer. Even smashing a bug under an electrical component could cause a failure.

    2. Re:The answer appears to be a yes. by xrayspx · · Score: 4, Funny

      Luckily, in the aerospace industry, there's no such thing as a "$5 wrench". Hell that was probably a $700 dirty rag.

    3. Re:The answer appears to be a yes. by gstoddart · · Score: 2

      Similarly, it's difficult to imagine a rag large enough to be useful but small enough to be left in a fuel line.

      How many satellites have you assembled? For me, it's zero.

      But, I've seen people clean the fuel line on RC helicopters ... we're talking about something about 1 inch square on the end of a little metal doo-hickey.

      It is not inconceivable that we're not talking about a big old smelly rag here.

      Why has Slashdot suddenly fallen into the trap of "I've never seen one so it can't possibly exist"? Seriously, I have no idea what is involved in putting a satellite together, but I usually think of dust-free chambers and people in fancy white suits; which means we're already well out of the mundane here.

      --
      Lost at C:>. Found at C.
    4. Re:The answer appears to be a yes. by galaad2 · · Score: 2, Informative

      bugs being smashed in electical components has already happened, lots of times in history.
      Here's one of the first properly documented cases of it, from 1947:

      http://www.history.navy.mil/photos/images/h96000/h96566k.jpg

      Photo #: NH 96566-KN (Color)

      The First "Computer Bug"

      Moth found trapped between points at Relay # 70, Panel F, of the Mark II Aiken Relay Calculator while it was being tested at Harvard University, 9 September 1947. The operators affixed the moth to the computer log, with the entry: "First actual case of bug being found". They put out the word that they had "debugged" the machine, thus introducing the term "debugging a computer program".
      In 1988, the log, with the moth still taped by the entry, was in the Naval Surface Warfare Center Computer Museum at Dahlgren, Virginia.

      Courtesy of the Naval Surface Warfare Center, Dahlgren, VA., 1988.

      NHHC Collection

      --
      root@127.0.0.1
    5. Re:The answer appears to be a yes. by roc97007 · · Score: 3, Informative

      I have assembled zero satellites. But back in my military contracting days, I did the electronics for several military applications and was present when satellites were built. The boosters on those things are fairly small, and the fuel lines tend not to be big enough to stuff into what we think of as a rag. Maybe a cleaning tool or some other implement. I think whomever wrote that was either lazy or didn't fully understand what they were writing about.

      > Why has Slashdot suddenly fallen into the trap of "I've never seen one so it can't possibly exist"?

      Have we so soon forgotten that us slashdotters come from a variety of backgrounds? For instance, legal articles are often responded to by actual lawyers in this group. There are actual astronomers, actual physicists, actual biologists, and I'm certain, actual rocket scientists, who read and participate in Slashdot. We're not all gamers living in our parent's basement. Although there are some.

      --
      Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
    6. Re:The answer appears to be a yes. by RingDev · · Score: 3, Informative

      Most starters aren't strong enough to bust up a wrench or socket. Take out a plug maybe, possibly bend a valve, but in all likelihood, the motor would turn the engine till contact and stop.

      That is assuming you are hitting the engin with the starter before hooking up the fuel and plugs. Which is usually a good idea to get the oil pump primed and heads lubricated firing it up.

      That said, I have a number of wrenches that could easily fit in a cylinder with the piston at BDC. A GM 350 for instance, has a 4" bore and 3.48" stroke. On the diagonal that gives you over 5 1/4" clearance at BDC, not including the combustion chamber in the head.

      9-11mm wrenches and 1/4" wrenches are common tools under the hood. Wiring brackets, trim plates, grounding lines, battery terminals, oil pan bolts, valve cover bolts, etc... They all fall into that size range.

      -Rick

      --
      "Most people in the U.S. wouldn't know they live in a tyrannical state if it walked up and grabbed their junk." - MyFirs
    7. Re:The answer appears to be a yes. by Dunbal · · Score: 2

      Why is everyone assuming a small diameter fuel line? It's a rocket engine not a 5cc internal combustion motor. You need a certain amount of gallons per minute for it to work.

      --
      Seven puppies were harmed during the making of this post.
    8. Re:The answer appears to be a yes. by AK+Marc · · Score: 2

      I believe the consensus is that the "rag" is a piece of cloth or cloth-like material designed small enough to fit in the line to clean it out. Most people think "cloth designed to be held by a hand" not "scrap designed to be held on the end of a rod for cleaning gun barrels and fuel lines." The wording was inexact and conjured an idea in most people that's inaccurate. Better wording "A small swatch of cleaning cloth is thought to have been left inside a fuel line" or something like that would have been more exact and less open to misunderstanding.

    9. Re:The answer appears to be a yes. by rts008 · · Score: 2

      You obviously have little or no experience with engines and wrenches.

      I have in my possession a canvas tool-pouch containing 12 ignition wrenches that can easily fit inside the cylinder, and has done so many times over the past 4 decades. That's 12 wrenches at a time, and the pouch that has fit totally inside of cylinders many times. (handy place to temporarily hold things)

      Somewhat in your defense, I will admit that there are few times you need that small of a wrench around an engine when the heads are removed, but saying "impossible" just highliights your ignorance.

      I personally have found many improbable items inside of engines over the decades, just as you here about surgery patients having post-operative complications from stuff left inside of them from the surgery.
      The 'ideal' or 'perfect' plan only exists on paper....throw a human in the mix, and you get Murpy's Law events sooner, or later.

      --
      Down With Slashdot BETA!!! I've been around the corner and seen the oliphant; you can only abuse me from your perspecti
    10. Re:The answer appears to be a yes. by JohnnyComeLately · · Score: 2

      Because it's hydrazine. Really nasty stuff no one wants to be shooting off on the ground. I read the link and I think whoever wrote the Air Force Magazine article either took liberties or talked to the only knucklehead around Space Command. For example, "We found things we hadn't seen before, such as a warm up period." Really? This is why every GPS satellite ever launched has pre-heaters for it's hydrazine thrusters? (They're called "cat bed heaters" if you're looking at Reaction Control System telemetry). He then quotes him to say, "The 50th SW sucked up the workload while doing normal operations," which was then contradicted by the finished statements, "Upon completion, it will be turned over to the 50th." (I'm paraphrasing.) LEO, or Launch and Early Orbit is normally not done by the same crews as on orbit ops. The 4 SOPS get's SCO, or Satellite Command Authority, after it's handed off by the group responsible for LEO has put it into operational orbit, and performed basic check-out to ensure it's mission capable. The 4 SOPS also has the unique situation of being augmented by a National Guard unit, the 148 SOPS, which also performs MILSTAR operations 24/7/365. So, the 50th has a "little" help :) when it comes to Milstar.

    11. Re:The answer appears to be a yes. by JohnnyComeLately · · Score: 2

      You have to be joking or you've forgotten what we're talking about. This is an apogee kick motor (to use GPS parlance) to take a 2 ton space vehicle to a circular 22,000 mile orbit. GPS, having half the orbit and half the weight, has an AKM which is not small. It's huge. It has to be due to the amount of firing it's intended to do. I tried Googling an image to show my point, but unfortunately, the AKM is on the "non-sexy" side of the SV and, hence, no photos. We're not talking about .5 pound stationkeeping thrusters.

      That said, yes I believe the author was undermined by a bad source, or at least the Air Force Magazine article I read which is cited. He is quoted as saying it was a surprise that hydrazine had a warm-up period, and made it sound as if the 50th was flying this bird alongside operable satellites. We've known for over 40 years hydrazine is more consistent when pre-warmed. This is why GPS fires up pre-heaters before every stationkeeping maneuver. Pre-heating gives more reliable, and predictable vectors.

  4. anything can take out a satellite by apcullen · · Score: 5, Funny

    It's hard. Any little thing that goes wrong will likely cause the whole thing not to work.
    That's why it's rocket science.

  5. Test Sequence? by Anonymous Coward · · Score: 4, Interesting

    Who puts an engine together without a test fire? Seems to me that some simple checks would have prevented a very big waste of funds and effort. I guess it won't be a total waste if they can learn from it.

    1. Re:Test Sequence? by blueg3 · · Score: 2

      Actually, the Hubble mirror isn't supposed to be flat, its shape is a particular function. It was actually manufactured exactly to spec, but the spec was wrong.

    2. Re:Test Sequence? by robot256 · · Score: 2

      Minor piece of historical correction: The mirror was supposed to parabolic, not flat, so measuring it was not quite a simple matter. It was not NASA but a subcontractor who did the actual grinding, and they used as a reference a defective measurement device when grinding it. Since they did not want to re-grind the mirror, they ignored the data from two functioning devices that said the mirror was flawed. NASA for its part did not perform sufficient quality control on the process to notice that only one device was used to measure it, and apparently did not test (or have the ability to test) the mirror independently. http://en.wikipedia.org/wiki/Hubble_Space_Telescope#Flawed_mirror

    3. Re:Test Sequence? by codegen · · Score: 3, Informative

      Actually, the Hubble mirror isn't supposed to be flat, its shape is a particular function. It was actually manufactured exactly to spec, but the spec was wrong.

      Actually,the hubble was spec'd to be a conic constant of p=-.0023, but was polished only to p=-.0139 (i.e. over hyperbolic). The error was due to a problem with the tester. The null reference element was out of position by just over a millimeter. The interesting thing is two other testers reported that the mirror was wrong, but they were ignored because they were not the 'primary' testing instrument. You are correct that it wasn't supposed to be flat, but it definitely wan't built to spec.

      --
      Atlas stands on the earth and carries the celestial sphere on his shoulders.
  6. Lots of failures there. by Lumpy · · Score: 5, Insightful

    Assembly failure - leave a rag.
    Inspection failure - did not check for rag.
    Pre-flight final inspection - still did not find the rag.

    Wow, complete failure all the way down the line from assembly to mating with the launch vehicle.

    --
    Do not look at laser with remaining good eye.
    1. Re:Lots of failures there. by Massacrifice · · Score: 5, Funny

      XXI century new space programs motto : It's failures all the way down, man!

      --
      -- Home is where you eat your heart out.
    2. Re:Lots of failures there. by geek · · Score: 4, Interesting

      It reminds me of those surgery horror stories where the surgeon or staff leaves behind clamps and sponges inside the persons body.

      Shit happens. All we can really do is our very best to try and prevent it, but ultimately, we're human and prone to mistakes.

    3. Re:Lots of failures there. by Overzeetop · · Score: 4, Interesting

      Actually pre-flight final won't catch that kind of thing; it's already buried in the system (and you don't fire thrusters on a flight unit prior to launch). This is likely one of those cases where a scrap of cleaning"rag" was torn off within the path in an area not visible at either end and went unnoticed. To save money, a visual of the system prior to final assembly was determined to be sufficient and the endoscope procedure was eliminated, saving several thousand dollars (combined on all the lines). Sure, in hindsight a compressed air test would have been sufficient, but it's a little late to play what-if now.

      --
      Is it just my observation, or are there way too many stupid people in the world?
    4. Re:Lots of failures there. by LWATCDR · · Score: 5, Informative

      You left out Slashdot summary failure.

      FTFA
      "On Oct. 24, AEHF-1 reached its originally planned orbit. Testing began soon afterward. The Air Force expects to bring the satellite into service in March. Meanwhile, two more AEHFs are slated to launch in 2012."

      They got it into the correct orbit over two months ago using the small thrusters.
      In other words...
      More sensationalistic headlines to get clicks and comments from the new Slashdot.
      Really? Oh and the answer is "no a dirty rag did not take out a 2 billion dollar commsat."
       

      --
      See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
    5. Re:Lots of failures there. by roc97007 · · Score: 4, Interesting

      > It reminds me of those surgery horror stories where the surgeon or staff leaves behind clamps and sponges inside the persons body.

      Funny you should mention that. I had emergency surgery last year for severe traumatic internal bleeding (won't bore you with the details -- or maybe I already have) and things happened so quickly that they did not have enough time for an instrument inventory. (Apparently it's someone's job to keep track of how many tools get used and then count them before final suture.) So after they got me stable they ran me back through x-ray to look for stuff. Didn't find anything, fortunately.

      But really -- it's not that much of a horror story, they just have to open you back up at some point to retrieve the objects. It's not something you want to have happen, but it's a fairly well known procedure. Horror stories to me are things like taking off the wrong limb [1] or prescribing catastrophically wrong medication.

      [1] Before I went in for knee surgery, the doctor gave me a sharpie and had me mark the correct knee. Just in case.

      --
      Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
    6. Re:Lots of failures there. by Baloroth · · Score: 2

      And of course that "underscores some of the weaknesses in U.S. space efforts." Actually, I would say it underscores the strength: they managed to fix the problem using ingenuity and scarce resources. Also, a "scrap of cloth" != "a rag". Calling it a rag implies someone just forgot a whole piece of cloth. A scrap of cloth implies it ripped or was otherwise accidentally and through no negligence (well, not gross negligence anyways, they may still have checked more carefully) deposited.

      --
      "None can love freedom heartily, but good men; the rest love not freedom, but license." --John Milton
    7. Re:Lots of failures there. by Overzeetop · · Score: 2

      The test was devised, mostly likely. There are hundreds - no, probably tens of thousands - of tests on this craft. You can probably trace every single raw material element back to where it was mined, refined, billeted, shipped, stored, manufactured, machined, binned, tagged, selected, gaged, installed, torqued, tested, and approved for flight. Every single time a human touches a part it costs $100 (well, that was a decade ago, it's probably $200 now).

      The anal retentiveness of the work flow on a satellite is mind boggling, and it's one of the reasons spacecraft are so fucking expensive. Over time, you try and weed out the tests which have traditionally had a very low failure rate in order to reduce the overall expense. A $1200 line scope doesn't sound like much. That scope on 4 lines is $5k. A pre-and post assembly scope makes it $10k. There are a thousand other parts that need to be checked with a single test pre and post assembly and now it's $10x10^6. That's after everything is fabricated (and each of those parts has been tested). How many of the 1300 nuts are you going to check with a go-no go gage? How many will you pull from the line and send out for destructive testing to verify chemical composition, temper condition? How many welds will you do no destructive testing on, and should you run both dye pen and magnetic? How about a few with ultrasound. Don't forget to check 10% with x-ray to check for oxides which can lead to fatigue failure that would be invisible with the dye pen test. And make sure that each welder makes a sample before he starts for each type and position of weld, then destructively test for signs of voids or other operator error.

      Now, we've barely scratched the surface, but when you add up all the tests, you start wondering if $500M in testing on a $2B payload might not be a bit high - why not take the ones that have NEVER come up as failures and evaluate which of those are likely to have the lowest probability of failure and eliminate them. There may be a couple million dollars. You're trading maybe 0.2% of the cost for a risk that may be out to the 8th or 9th decimal place in reliability.

      As much shit as I'm willing to give contractors in a lot of cases, the guys on the floor are emotionally invested in the success of a big mission. These tests will go back in, and the next one will be more expensive (well, not really - the engineering is done, the next one will cost less, but not an insignificant amount). It's impossible to account for every eventuality, and also economically irresponsible. At some point, there will be risk. This time, risk won.

      --
      Is it just my observation, or are there way too many stupid people in the world?
  7. Foreign object debris seems to be common... by Richard_at_work · · Score: 4, Interesting

    At least one of the recent Soyuz failures was put down to a similar issue - debris left in a fuel line by a worker.

    1. Re:Foreign object debris seems to be common... by DrBoumBoum · · Score: 2
      I've found this story about US Atlas program in 1993:

      Then, three years later, it was the Atlas program’s turn again. On March 25, 1993, an Atlas 1 lifted off from SLC-36B at Cape Canaveral AFS, carrying the first of the US Navy’s new UHF Follow-On communication satellites. The launch proved to an inauspicious start to the new program.

      A mere 22 seconds after liftoff the vehicle’s sustainer engine began to lose thrust, ultimately reaching only 65% of its nominal thrust level at T+103 sec. The Centaur second stage performed normally, but was inadequate to the task of making up for the low performance of the sustainer. The payload ended up in an orbit far below the desired geosynchronous transfer orbit. The spacecraft used its own onboard propulsion system to climb to a higher orbit, but one that still proved to be too low to meet mission requirements.

      Analysis showed that the sustainer thrust decay was due to a simple problem. The Atlas sustainer engine thrust level was controlled by a regulator that was adjusted by turning a screw. A set screw was to be tightened to ensure that the adjustment screw did not move due to in-flight vibration, and that had not been done properly. The result was another fatal Oops!

  8. The result of a GAO audit? by JoeMerchant · · Score: 2

    So, somebody can't come up with the used rag disposal accounting paperwork and the GAO concludes that it must have been left inside?

    I mean, this kind of thing is good for sponges during surgery, why not satellite assembly?

  9. Send up some Midol? by rts008 · · Score: 5, Funny

    So, the problem is the satellite is 'on the rag'?

    --
    Down With Slashdot BETA!!! I've been around the corner and seen the oliphant; you can only abuse me from your perspecti
  10. Speculation, not fact. by pla · · Score: 4, Insightful

    FTA: "They didn’t know it at the time, but a fuel line had become clogged. The blockage “was most likely caused by a small piece of cloth inadvertently left in the line during the manufacturing process,” according to the Government Accountability Office." (bolding mine).

    So no, we don't know that a dirty rag caused a two billion dollar satellite to fail. We think a fuel line became clogged, and some government bean-counter pulled the dirty-rag hypothesis straight out of their derriere so they could sign off on this one and go home.

    1. Re:Speculation, not fact. by hrvatska · · Score: 2

      FTA: "They didn’t know it at the time, but a fuel line had become clogged. The blockage “was most likely caused by a small piece of cloth inadvertently left in the line during the manufacturing process,” according to the Government Accountability Office." (bolding mine). So no, we don't know that a dirty rag caused a two billion dollar satellite to fail. We think a fuel line became clogged, and some government bean-counter pulled the dirty-rag hypothesis straight out of their derriere so they could sign off on this one and go home.

      The GAO was probably basing its conclusion on statements from Lockheed itself. According to this it was Lockheed that concluded the problem was some cleaning material left in the line.

      "It should not have happened,” Deputy Under Secretary of the Air Force for Space Programs Richard McKinney said. “It was a quality mistake and we took steps to make sure it does not happen again,” he said. “It was obviously a very serious error.”

      “It appears that there was a blockage in one of the fuel lines,” McKinney said. Lockheed thinks “it was caused by some cleaning material that was used in a line that was not properly vacated when they went through production.”

    2. Re:Speculation, not fact. by pla · · Score: 2

      You mean the same way you pulled that out of your ass?

      Please describe to me the process by which you would prove that a fuel line currently in geostationary orbit 24,000 miles above the surface of the Earth has a dirty rag (specifically - As opposed to a dead mouse or a styrofoam peanut, for example) blocking it, without taking it apart and finding said rag.

      Yeah, thought so, "dipshit".

  11. Heading hyperbole by biometrizilla · · Score: 5, Informative

    Already been established that they were able to overcome the rag and get the satellite into a functional orbit where it can fulfill its mission objective. http://www.spaceflightnow.com/atlas/av019/120103rescue.html

  12. Glad they didn't go to a backup! by wisebabo · · Score: 4, Insightful

    "Finally, it speaks to the size and age of the U.S. space arsenal that the Air Force felt it had no choice but to rescue AEHF-1 instead of replace it with a back-up spacecraft. 'The asset inventory is getting so tight that they spent months limping the heap to its proper orbit,' the insider lamented."

    Look guys, before you throw away (replace with a backup) a $2 Billion satellite, I damn well hope you try some pretty heroic measures. Those are my tax dollars in (the wrong) orbit! So I'm very glad you didn't have (to use) a backup satellite.

    Anyway, does anyone know if the low power thrusters which were eventually used to put this satellite into the correct orbit used the same fuel tank as the clogged thruster? Otherwise 1) I'm very surprised they had enough fuel to get there and 2) they would probably have very little left to last the lifetime of the mission. So let's hope that all the thrusters used a central (hydrazine?) fuel tank and there's plenty left.

    Space is hard and while the U.S. program has certainly had its ups and downs at least it hasn't seen the near total collapse as what happened to the Ruskies. They had quite a bad year last year and that blogger walking around their factory just exposed their problems more. If Mars is going to be a "Red" planet it will because of China not Russia.

    1. Re:Glad they didn't go to a backup! by LoveMuscle · · Score: 2

      According to this:

      http://spaceflightnow.com/atlas/av019/111009.html

      They used the hall effect thrusters instead of the hydrazine/nitrogen tetraoxide engine. The hall effect thrusters run on xenon and electricity, so NO they did not use the same fuel source. The hall effect thrusters have a specific impulse of ~8000s instead of the ~300s for hydrazine, so they are insanely fuel efficient, but extremely low thrust. (1/4N vs ~450N for the main engine)..

  13. RAG?? by Anonymous Coward · · Score: 2, Informative

    A Dirty Rag? C'mon - RTFA! "The blockage 'was most likely caused by a small piece of cloth inadvertently left in the line during the manufacturing process,' according to the Government Accountability Office."

    That could mean a tiny fragment of fabric. It's not like they put a rag in the gas tank to keep gas from leaking out. sheesh.

  14. Check the logs? by Maximum+Prophet · · Score: 4, Funny

    So why do they not check the forms before launching the satellite into orbit?

    --
    All ideas^H^H^H^H^Hprocesses in this post are Patent Pending. (as well as the process of patenting all postings)
    1. Re:Check the logs? by hedwards · · Score: 4, Insightful

      Probably the same reason why things end up being left inside of patients. Accidents happen, even if it's something that should never happen because it was on the checklist.

    2. Re:Check the logs? by SplashMyBandit · · Score: 2

      Actually, stuff gets left in patients because surgeons don't use checklists. Apparently think they have fantastic memories and no-one who is not a higher-qualified surgeon is gonna tell them what to do. However, a recent study was done where surgeons took a left from pilot's books and actually started using checklists - the rate of post-operative complications plummeted. The use of checklists has not yet reached widespread use in surgery, AFAIK. I hope it does soon.

  15. Weaponize it by Ambitwistor · · Score: 2

    We must be increasingly on the alert to prevent our enemies from taking over our satellite fuel lines, thus knocking out our military communications. Mr. President, we must not allow a dirty rag gap!

  16. One would think by ThatsNotPudding · · Score: 2

    just like in invasive surgery, there would be a known count of 'sponges' and after buttoning up, they had all better be accounted for.

  17. Blowing out lines with air... by Kamiza+Ikioi · · Score: 2

    Billions of dollars in technology, but nobody with an air nozzle hooked up to an air compressor found at any car mechanic's shop to blow out a fuel line.

    --
    I8-D
    1. Re:Blowing out lines with air... by netwarerip · · Score: 3, Funny

      They had one, but none of the engineers had 2 quarters to put in the machine.

  18. aliens by j00r0m4nc3r · · Score: 2

    Why do they dismiss the possibility of alien parasites that look like rags and feed on satellite engine fuel?

  19. Bottle in the car by satuon · · Score: 2

    Reminds me about all those stories of bottles put inside cars during assembly. Here's a funny one (albeit fictional):

    A man goes to a car dealership one day after inheriting a good deal of money (or after a great business deal, whatever -- he has a lot of money somehow). After looking around the lot, he picks out the nicest, newest, fanciest, most expensive car he can. He pays cash up front and drives out of the dealership in the new car.

    On his way home, he starts hearing a rattling sound -- something must be wrong. So he turns around and goes right back to the dealer. The dealer is of course very sorry, and offers to either fix the car or let the man take a different one while they order a replacement. The man really wants the car, so he just has the guy fix it. Two hours later, the mechanics give the car back, saying they couldn't find a thing wrong with it. The man is a bit wary, but he drives home. Whatever the rattle is, it has stopped.

    A day or so later, the rattle starts again. He takes it to the dealership, and they still can't find anything wrong with it. This continues for a number of weeks -- sometimes the rattle even goes away on its own. Anyway, after nearly two months of it, the dealer is very upset -- he doesn't want to get a bad reputation. So he orders a replacement and exchanges it with the man for the malfunctional car.

    Then he orders the mechanics in the shop to do a complete tear-down to figure out the problem. They begin taking the car apart, piece by piece, but they can't find anything -- until they take apart the door. Inside, they find a piece of metal pipe, along with a note. Written on the note, in a scrawling, worker's hand is: "So, you finally found the rattle, you rich son-of-a-bitch."

  20. my uncle (cancer patient) died from forgotten rags by tomweeks · · Score: 2

    My uncle was in cancer remission for >20ys when the US-VA went in to fix a simple abdominal hernia. After they sewed him up, he was okay for a couple of weeks, but then got very sick. An infection was ravaging his body and the doctors could not localize the infection. After months of antibiotics, they found the rags and removed them and scooped out all the necrosing tissue. By this time, his immune system had sunken so low that the cancer was able to make a come back which ended up killing him (after months of chemo).

    All this to say.. forgotten rags (and implements) are a pretty significant problem in many disciplines (not just fuel lines), and why many operating rooms now have a "time out" materials count in pre and post-op. Maybe launch pad protocols should adopt the same.

    Tweeks

  21. CHECKLIST by bussdriver · · Score: 3, Informative

    The airlines did it and improved by amazing amounts (nobody remembers how bad it was) and the things were much less complex to fly back in those days; the pilots were insulted by it as well. CHECKLISTS WORK.

    Something that important should involve multiple checklists; to error is human no matter how good and smart you are. Doctors are the most arrogant pricks I've ever met so they'll put up a huge fight and have a hard time admitting it when the error rate goes down by half. It likely would go down by half; that is how badly it is needed.

    Nurses too... a friend of mine fought off his nurse violently (as much as he had strength post op) she had to call people in to hold him down and sedate him and luckily somebody heard his screams and READ the chart and realized she had the wrong person! he would have died and without a proper autopsy the cause wouldn't have been known. Mistakes killed my father too. Checklists must be mandatory by law like the pilots who have no issue with them today.