Slashdot Mirror
Incorrectly Built SLS Welding Machine To Be Rebuilt
schwit1 writes A giant welding machine, built for NASA's multi-billion dollar Space Launch System (SLS), has to be taken apart and rebuilt because the contractor failed to reinforce the floor, as required, prior to construction: "Sweden's ESAB Welding & Cutting, which has its North American headquarters in Florence, South Carolina, built the the roughly 50-meter tall Vertical Assembly Center as a subcontractor to SLS contractor Boeing at NASA's Michoud Assembly Facility in New Orleans.
ESAB was supposed to reinforce Michoud's floor before installing the welding tool, but did not, NASA SLS Program Manager Todd May told SpaceNews after an April 15 panel session during the 31st Space Symposium here. As a result, the enormous machine leaned ever so slightly, cocking the rails that guide massive rings used to lift parts of the 8.4-meter-diameter SLS stages The rings wound up 0.06 degrees out of alignment, which may not sound like much, "but when you're talking about something that's 217 feet [66.14 meters] tall, that adds up," May said.
Asked why ESAB did not reinforce the foundation as it was supposed to, May said only it was a result of "a miscommunication between two [Boeing] subcontractors and ESAB."
It is baffling how everyone at NASA, Boeing, and ESAB could have forgotten to do the reinforcing, even though it was specified in the contract. It also suggests that the quality control in the SLS rocket program has some serious problems.
ESAB was supposed to reinforce Michoud's floor before installing the welding tool, but did not, NASA SLS Program Manager Todd May told SpaceNews after an April 15 panel session during the 31st Space Symposium here. As a result, the enormous machine leaned ever so slightly, cocking the rails that guide massive rings used to lift parts of the 8.4-meter-diameter SLS stages The rings wound up 0.06 degrees out of alignment, which may not sound like much, "but when you're talking about something that's 217 feet [66.14 meters] tall, that adds up," May said.
Asked why ESAB did not reinforce the foundation as it was supposed to, May said only it was a result of "a miscommunication between two [Boeing] subcontractors and ESAB."
It is baffling how everyone at NASA, Boeing, and ESAB could have forgotten to do the reinforcing, even though it was specified in the contract. It also suggests that the quality control in the SLS rocket program has some serious problems.
"and not micromanage it". That's the rub. The micromanaging, the reporting and compliance costs, can be over 50% of the cost for some federal contracts, but most of the time that's required by thousands upon thousands of pages of federal law. When you have a comoany that knows how to do a certain thing , aka one of those evil corporations, getting hired by the federal government, some people want to do a lot of paperwork and stuff to keep track of what's going on, and other people go crazy with it. The organization I work for used to do a lot of federal contracts. We quit and now just do state contracts for states that are reasonable.
Still other people added a bunch of requirements for federal contracting that aren't really relevant to the project. For example, how many black women work for each of your major suppliers? How much do your interns make? Are all of the web pages and documentation you've ever made fully accessible to people who are both blind and deaf?
We quit dealing with the feds and certain states because it's just not worth it. It would cost SPACEX five times as much to build a federally-contracted rocket than it costs to build their own.
Age old story of outsourcing - you still need to retain enough people to watch the contractors so they can't cut corners on the expensive bits.
One blatant example I saw was with non destructive testing of welds in high pressure pipework leading to portions of a turbine in a coal fired power station. At those welds it was done by spraying on thing white paint, using a magnet and spraying on a fluid with suspended magnetic "dust" that would collect wherever defects disrupted the magnetic field. Access was a bit tight so the contractors tested the top of the pipes and they ran the magnet around the bottom of the pipe without looking at it so that some scratches would be left to show that the magnet had been used. The lazy pricks were caught doing that so we had to send someone along as an observer and make them do a couple of weeks worth of work over again, because with their scratch trick we had no way of knowing is any inspection had actually been done or not.
So MBA types - that person standing off to the side not doing anything during a concrete pour may be there solely to reduce fuckups due to dishonest contractors.
If we're doing something important, like killing Hitler, or trying to beat the Commies to the moon, federal procurement can be remarkably efficient. Clear goals, and the stated willingness to accept some waste as long as the job's done, can do that.
Unfortunately since about the mid-1970s (Watergate you say?!), approximately zero "waste," of any kind is tolerated on any federal project, as this is "profiteering" or "wasting the people's money," so a lot of contractor time is spent on compliance. This makes the process incredibly loss-averse, and probably too risk-averse to actually accomplish anything.
The reality is that Elon Musk is able to do a good job, because he can destroy two or three recovery barges in a row and he doesn't have to explain it to anybody but his accountant. If the SLS had only one slip-up like that there would be a bloodbath of firings, senate hearings, press conferences with the President, and maybe the entire program might be scrubbed. Back in the late 50s NASA screwed up these kinds of operations all the time, but the American people tolerated it because of the Cold War. Nowadays the budget is so tight and public accountability is so fierce that frigging welding assembly subcontractors are apparently front-page news. We probably built and destroyed five facilities on the scale of this thing during Apollo and nobody batted an eye at the expense.
Don't blame me, I voted for Baltar.
Thanks for that.
These days, though, "lax testing standards" is in the eye of the beholder. I work for a Japanese manufacturer with a world-renowned reputation for quality.
In the US, having the word "quality" in your job title is a career death sentence. In this Japanese company, it means that you are the créme de la créme.
They look at testing as "black box monkey testing" at the end of the process. This is what they have done for multiple decades. They have Excel checklists that are thousands of lines long. If even one of those lines is not checked, or is anomalous, the whole shooting match comes to a screeching halt until said line is graced with a green check.
I am trying to implement an inline "process quality" to their software development. You know, TDD, CI, CD and whatnot.
My. God. It's damn near impossible.
For one thing, they have a culture that deliberately sets up an antagonistic relationship between Quality and Engineering. Quality generally has more power. Engineering is considered to be a bunch of "yahoos," bent on degrading the Holy Quality.
When Engineering managers (like me) try to suggest quality measures; even though these are not wild, "cutting edge" things, we are routinely dismissed and ignored, because we are obviously trying to avoid work and introduce bad quality.
The simple fact of the matter is, that introducing the techniques I'm talking about could have a revolutionary effect on our quality. It would drastically reduce the cost of our production, and would certainly reduce (possibly to 0), the number of "time bomb" bugs that tend to slip through those massive checklists and explode in the face of influential people with high Twitter follower accounts.
I strongly suspect that NASA has an extremely similar culture. The only way they can grok "quality," is through their 1950s-era "black box at the end" testing methodology.
This has some real benefits (Read this to see what can happen when you have super-redundant black box testing and prototyping).
However, it is screamingly -UNBELIEVABLY- expensive and time-consuming (read: "expensive"). It also doesn't guarantee quality. Nothing really does.
In the case of Apollo 13, NASA's anal prototyping and simulation exercises saved the astronauts. However, the preferred outcome would have been that the oxygen tank not have exploded in the first place. A lot more boring.
I'm not sure that your statement was fair.
Musk is an engineer. Quality folks see engineers as "out-of-control cowboys."
I'm getting very, very sick of this highly insulting, and completely inaccurate portrayal.