Slashdot Mirror
Incorrectly Built SLS Welding Machine To Be Rebuilt
schwit1 writes A giant welding machine, built for NASA's multi-billion dollar Space Launch System (SLS), has to be taken apart and rebuilt because the contractor failed to reinforce the floor, as required, prior to construction: "Sweden's ESAB Welding & Cutting, which has its North American headquarters in Florence, South Carolina, built the the roughly 50-meter tall Vertical Assembly Center as a subcontractor to SLS contractor Boeing at NASA's Michoud Assembly Facility in New Orleans.
ESAB was supposed to reinforce Michoud's floor before installing the welding tool, but did not, NASA SLS Program Manager Todd May told SpaceNews after an April 15 panel session during the 31st Space Symposium here. As a result, the enormous machine leaned ever so slightly, cocking the rails that guide massive rings used to lift parts of the 8.4-meter-diameter SLS stages The rings wound up 0.06 degrees out of alignment, which may not sound like much, "but when you're talking about something that's 217 feet [66.14 meters] tall, that adds up," May said.
Asked why ESAB did not reinforce the foundation as it was supposed to, May said only it was a result of "a miscommunication between two [Boeing] subcontractors and ESAB."
It is baffling how everyone at NASA, Boeing, and ESAB could have forgotten to do the reinforcing, even though it was specified in the contract. It also suggests that the quality control in the SLS rocket program has some serious problems.
ESAB was supposed to reinforce Michoud's floor before installing the welding tool, but did not, NASA SLS Program Manager Todd May told SpaceNews after an April 15 panel session during the 31st Space Symposium here. As a result, the enormous machine leaned ever so slightly, cocking the rails that guide massive rings used to lift parts of the 8.4-meter-diameter SLS stages The rings wound up 0.06 degrees out of alignment, which may not sound like much, "but when you're talking about something that's 217 feet [66.14 meters] tall, that adds up," May said.
Asked why ESAB did not reinforce the foundation as it was supposed to, May said only it was a result of "a miscommunication between two [Boeing] subcontractors and ESAB."
It is baffling how everyone at NASA, Boeing, and ESAB could have forgotten to do the reinforcing, even though it was specified in the contract. It also suggests that the quality control in the SLS rocket program has some serious problems.
The article does not mention where the cost of this error is going to fall. This seems like an important detail. On a sufficently complex project, one of the bevy of subcontractors fucking something up isn't a huge surprise; but I would be very, very, disappointed if NASA wasn't able to contract sufficiently vigorously to make the vendor eat the cost of delivering the goods as specified, rather than paying them for their effort no matter how well or badly they do.
I'm pretty sure that letting the contractors bilk NASA is the point of the exercise at this stage. The SLS isn't referred to as the Senate Launch System for nothing.
"and not micromanage it". That's the rub. The micromanaging, the reporting and compliance costs, can be over 50% of the cost for some federal contracts, but most of the time that's required by thousands upon thousands of pages of federal law. When you have a comoany that knows how to do a certain thing , aka one of those evil corporations, getting hired by the federal government, some people want to do a lot of paperwork and stuff to keep track of what's going on, and other people go crazy with it. The organization I work for used to do a lot of federal contracts. We quit and now just do state contracts for states that are reasonable.
Still other people added a bunch of requirements for federal contracting that aren't really relevant to the project. For example, how many black women work for each of your major suppliers? How much do your interns make? Are all of the web pages and documentation you've ever made fully accessible to people who are both blind and deaf?
We quit dealing with the feds and certain states because it's just not worth it. It would cost SPACEX five times as much to build a federally-contracted rocket than it costs to build their own.
Rather the opposite. WIth the "libtards" truly in charge, there would be no outsourcing and subcontracting, and NASA would hire people to build things themselves.
The republicans are the ones that demand outsourcing and paperwork that often equals half the total costs. Because heavens forbid if a government agency did something that private companies could do. That is considered anticompetitive theft by the right. Which is why NASA can't do much themselves anymore, and get less bang for the buck.
Age old story of outsourcing - you still need to retain enough people to watch the contractors so they can't cut corners on the expensive bits.
One blatant example I saw was with non destructive testing of welds in high pressure pipework leading to portions of a turbine in a coal fired power station. At those welds it was done by spraying on thing white paint, using a magnet and spraying on a fluid with suspended magnetic "dust" that would collect wherever defects disrupted the magnetic field. Access was a bit tight so the contractors tested the top of the pipes and they ran the magnet around the bottom of the pipe without looking at it so that some scratches would be left to show that the magnet had been used. The lazy pricks were caught doing that so we had to send someone along as an observer and make them do a couple of weeks worth of work over again, because with their scratch trick we had no way of knowing is any inspection had actually been done or not.
So MBA types - that person standing off to the side not doing anything during a concrete pour may be there solely to reduce fuckups due to dishonest contractors.
If we're doing something important, like killing Hitler, or trying to beat the Commies to the moon, federal procurement can be remarkably efficient. Clear goals, and the stated willingness to accept some waste as long as the job's done, can do that.
Unfortunately since about the mid-1970s (Watergate you say?!), approximately zero "waste," of any kind is tolerated on any federal project, as this is "profiteering" or "wasting the people's money," so a lot of contractor time is spent on compliance. This makes the process incredibly loss-averse, and probably too risk-averse to actually accomplish anything.
The reality is that Elon Musk is able to do a good job, because he can destroy two or three recovery barges in a row and he doesn't have to explain it to anybody but his accountant. If the SLS had only one slip-up like that there would be a bloodbath of firings, senate hearings, press conferences with the President, and maybe the entire program might be scrubbed. Back in the late 50s NASA screwed up these kinds of operations all the time, but the American people tolerated it because of the Cold War. Nowadays the budget is so tight and public accountability is so fierce that frigging welding assembly subcontractors are apparently front-page news. We probably built and destroyed five facilities on the scale of this thing during Apollo and nobody batted an eye at the expense.
Don't blame me, I voted for Baltar.
Yup. I've had to deal with contractors in the physical realm (cabling and other physical infrastructure) and in the logical realm (switch and router programming) and with the cabling if we weren't constantly inspecting them we got absolute crap. Punched the wrong color pattern (sorry we're T568A, but deal with it), leaving out drops, leaving out service loops, leaving out cabling supports, attempting to pass-off PVC in a plenum airspace, attempting to use lesser jacks (we called for Leviton for a reason dammit!), and lazy, lazy bastards that couldn't comprehend no zip ties.
On the switch side, they're getting $1200 per switch to program and rack them. We started calling it $300 a screw but that suddenly became not-funny after we found them using 10-32 screws in 12-24 racks, or leaving half of the screws out altogether, and that's before dealing with the crappy programming that I've had to go back and fix. It's so much fun when they never test the user VLAN, but the management VLAN trunks through, so the switch MUST work, right?
Anyway, Contractors are lazy bastards that will do as little as possible to satisfy the contract, and when called on their mistakes will try to weasel out of them. A NOAA satellite fell over at the manufacturer's assembly building because the morons forgot to check all of the bolts in the assembly hoist, and they got away with simply 'making no profit', when they should have paid the entire cost of the SNAFU. Perkin Elmer, who ground the Hubble mirror wrong, should have had to pay for the cost to repair it IN SPACE or should have suffered the corporate death penalty, had their charter revoked.
Screw contractors. They'll screw you.
Do not look into laser with remaining eye.
Actually, that's only sort-of true. I started with MCC interim release, but couldn't get it to work properly. So then I spent a few days downloading SLS and it worked just fine - well, as good as you could expect with only 4MB of ram. But I didn't notice any alignment issues, and I wasn't instructed to reinforce the floor so I didn't. I had problems with overheating during compilation though, which I fixed by a powerful floor fan pointed at the air intake of the PC. I later fixed this more gracefully with a home-made triple-sized heat sink. Maybe that's what NASA should do, build a giant heat sink onto it.
Since I'm working in a large organization I have come to realize that the amount of documentation in many projects is huge - often so large that essential key information is masked away, or right out FUBARed.
It's also not uncommon that the customer requirements are "interpreted" by people with no technical knowledge whatsoever and they have a tendency to eradicate information that they think is "too technical", or information that they think drives unnecessary cost. Some people also have a tendency to rename things to a semantic that is to common people fuzzy. Even obfuscation occurs. At the same time documents are filled with a large number of pages listing old or discarded alternatives.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
Asked why ESAB did not reinforce the foundation as it was supposed to, May said only it was a result of "a miscommunication between two [Boeing] subcontractors and ESAB."
Bullshit. The reason is they thought they could get away with it.
It was totally the last CEO's fault. The $50million golden parachute was money well spent getting rid of that guy so we could hire someone competent.
Sleep your way to a whiter smile...date a dentist!
ESAB is a Swedish company. What use is it to NASA to dote largess on a Swedish welding firm?
I'm actually rather disappointed with ESAB here. I have one of their MIG welders from the 1960s and it still works; they're a respectable name.
I feel bad for NASA mind you, in that I don't think many of their problems are their own. They get all sorts of legacy systems forced upon them due to political reasons ("You can't do decision X that would be more efficient because 1000 people in my district would lose their jobs"), they never get the funding to engineer new things from scratch based on lessons learned, etc. I do wonder, mind you, whether their heavy reliance on external contractors is something they could reform.
*Kid Rock runs for Senate* Democrats: We must run Kid Scissors.
The contractors that NASA has used have squandered and bilked NASA for all they are worth - If you give the goal of what the SLS does to SpaceX - (and not micromanage it) Elon Musk will have it completed in a much shorter amount of time.
I work for NASA. While I don't have direct involvement with the commercial crew program, major portions of it go on around me so I am familiar with how it works. (There was a 3 hour briefing on the various aspects of it in our department a few weeks back.)
SpaceX does some cool stuff but the biggest reason they're able to do things so cheaply is that they have pretty lax testing standards. They lack the analysis experts NASA has; Lockheed does too, but have more experienced people. Everyone at SpaceX seems to be under 30 (really) and it's a bit of an edgy place to work thanks to Elon Musk's propensity to walk through the workplace, question people at random, and occasionally fire them if he doesn't like what he sees. In fact, a lot of the analysis that should be done on SpaceX stuff is done by NASA because 1) SpaceX doesn't want to do it, and 2) they don't really know how.
I wouldn't go so far as to say that SpaceX is an accident waiting to happen, but their standards are more like the Russian ones. And their attitude toward higher standards is pretty much "whatever."
If you're wondering, the reason SpaceX got so much less money than Boeing in the recent commercial crew contract is that they asked for a lot less. They don't have as much experience as Boeing and it is felt they significantly underbid the contract due to not having done a project of that type before.
Whenever the blame is on "miscommunications", the real cause is likely poor project management, both on the owner and contractor/subcontractor side. The owner needs a good PM in place to ensure their critical requirements are being met. The constructor PM needs to ensure they understand and have a plan in place to meet all requirements. The blame likely lies between those two roles.
Thanks for that.
These days, though, "lax testing standards" is in the eye of the beholder. I work for a Japanese manufacturer with a world-renowned reputation for quality.
In the US, having the word "quality" in your job title is a career death sentence. In this Japanese company, it means that you are the créme de la créme.
They look at testing as "black box monkey testing" at the end of the process. This is what they have done for multiple decades. They have Excel checklists that are thousands of lines long. If even one of those lines is not checked, or is anomalous, the whole shooting match comes to a screeching halt until said line is graced with a green check.
I am trying to implement an inline "process quality" to their software development. You know, TDD, CI, CD and whatnot.
My. God. It's damn near impossible.
For one thing, they have a culture that deliberately sets up an antagonistic relationship between Quality and Engineering. Quality generally has more power. Engineering is considered to be a bunch of "yahoos," bent on degrading the Holy Quality.
When Engineering managers (like me) try to suggest quality measures; even though these are not wild, "cutting edge" things, we are routinely dismissed and ignored, because we are obviously trying to avoid work and introduce bad quality.
The simple fact of the matter is, that introducing the techniques I'm talking about could have a revolutionary effect on our quality. It would drastically reduce the cost of our production, and would certainly reduce (possibly to 0), the number of "time bomb" bugs that tend to slip through those massive checklists and explode in the face of influential people with high Twitter follower accounts.
I strongly suspect that NASA has an extremely similar culture. The only way they can grok "quality," is through their 1950s-era "black box at the end" testing methodology.
This has some real benefits (Read this to see what can happen when you have super-redundant black box testing and prototyping).
However, it is screamingly -UNBELIEVABLY- expensive and time-consuming (read: "expensive"). It also doesn't guarantee quality. Nothing really does.
In the case of Apollo 13, NASA's anal prototyping and simulation exercises saved the astronauts. However, the preferred outcome would have been that the oxygen tank not have exploded in the first place. A lot more boring.
I'm not sure that your statement was fair.
Musk is an engineer. Quality folks see engineers as "out-of-control cowboys."
I'm getting very, very sick of this highly insulting, and completely inaccurate portrayal.