Slashdot Mirror
Incorrectly Built SLS Welding Machine To Be Rebuilt
schwit1 writes A giant welding machine, built for NASA's multi-billion dollar Space Launch System (SLS), has to be taken apart and rebuilt because the contractor failed to reinforce the floor, as required, prior to construction: "Sweden's ESAB Welding & Cutting, which has its North American headquarters in Florence, South Carolina, built the the roughly 50-meter tall Vertical Assembly Center as a subcontractor to SLS contractor Boeing at NASA's Michoud Assembly Facility in New Orleans.
ESAB was supposed to reinforce Michoud's floor before installing the welding tool, but did not, NASA SLS Program Manager Todd May told SpaceNews after an April 15 panel session during the 31st Space Symposium here. As a result, the enormous machine leaned ever so slightly, cocking the rails that guide massive rings used to lift parts of the 8.4-meter-diameter SLS stages The rings wound up 0.06 degrees out of alignment, which may not sound like much, "but when you're talking about something that's 217 feet [66.14 meters] tall, that adds up," May said.
Asked why ESAB did not reinforce the foundation as it was supposed to, May said only it was a result of "a miscommunication between two [Boeing] subcontractors and ESAB."
It is baffling how everyone at NASA, Boeing, and ESAB could have forgotten to do the reinforcing, even though it was specified in the contract. It also suggests that the quality control in the SLS rocket program has some serious problems.
ESAB was supposed to reinforce Michoud's floor before installing the welding tool, but did not, NASA SLS Program Manager Todd May told SpaceNews after an April 15 panel session during the 31st Space Symposium here. As a result, the enormous machine leaned ever so slightly, cocking the rails that guide massive rings used to lift parts of the 8.4-meter-diameter SLS stages The rings wound up 0.06 degrees out of alignment, which may not sound like much, "but when you're talking about something that's 217 feet [66.14 meters] tall, that adds up," May said.
Asked why ESAB did not reinforce the foundation as it was supposed to, May said only it was a result of "a miscommunication between two [Boeing] subcontractors and ESAB."
It is baffling how everyone at NASA, Boeing, and ESAB could have forgotten to do the reinforcing, even though it was specified in the contract. It also suggests that the quality control in the SLS rocket program has some serious problems.
is a penny burned
“He’s not deformed, he’s just drunk!”
The contractors that NASA has used have squandered and bilked NASA for all they are worth - If you give the goal of what the SLS does to SpaceX - (and not micromanage it) Elon Musk will have it completed in a much shorter amount of time.
... is ten dollars on the shop floor and a hundred dollars in the field.
Looks like they tried to save eleven dollars and got caught out!
The article does not mention where the cost of this error is going to fall. This seems like an important detail. On a sufficently complex project, one of the bevy of subcontractors fucking something up isn't a huge surprise; but I would be very, very, disappointed if NASA wasn't able to contract sufficiently vigorously to make the vendor eat the cost of delivering the goods as specified, rather than paying them for their effort no matter how well or badly they do.
Its nearly 3 inces. What was the precision of the Saturn 5?
"and not micromanage it". That's the rub. The micromanaging, the reporting and compliance costs, can be over 50% of the cost for some federal contracts, but most of the time that's required by thousands upon thousands of pages of federal law. When you have a comoany that knows how to do a certain thing , aka one of those evil corporations, getting hired by the federal government, some people want to do a lot of paperwork and stuff to keep track of what's going on, and other people go crazy with it. The organization I work for used to do a lot of federal contracts. We quit and now just do state contracts for states that are reasonable.
Still other people added a bunch of requirements for federal contracting that aren't really relevant to the project. For example, how many black women work for each of your major suppliers? How much do your interns make? Are all of the web pages and documentation you've ever made fully accessible to people who are both blind and deaf?
We quit dealing with the feds and certain states because it's just not worth it. It would cost SPACEX five times as much to build a federally-contracted rocket than it costs to build their own.
Rather the opposite. WIth the "libtards" truly in charge, there would be no outsourcing and subcontracting, and NASA would hire people to build things themselves.
The republicans are the ones that demand outsourcing and paperwork that often equals half the total costs. Because heavens forbid if a government agency did something that private companies could do. That is considered anticompetitive theft by the right. Which is why NASA can't do much themselves anymore, and get less bang for the buck.
Age old story of outsourcing - you still need to retain enough people to watch the contractors so they can't cut corners on the expensive bits.
One blatant example I saw was with non destructive testing of welds in high pressure pipework leading to portions of a turbine in a coal fired power station. At those welds it was done by spraying on thing white paint, using a magnet and spraying on a fluid with suspended magnetic "dust" that would collect wherever defects disrupted the magnetic field. Access was a bit tight so the contractors tested the top of the pipes and they ran the magnet around the bottom of the pipe without looking at it so that some scratches would be left to show that the magnet had been used. The lazy pricks were caught doing that so we had to send someone along as an observer and make them do a couple of weeks worth of work over again, because with their scratch trick we had no way of knowing is any inspection had actually been done or not.
So MBA types - that person standing off to the side not doing anything during a concrete pour may be there solely to reduce fuckups due to dishonest contractors.
I hate watching Grand Designs and then there's someone who has ordered three glass insulation windows from Sweden (most likely the case, it may just have happened like two times or whatever but still!) and then it either takes forever or there's wrong measurements / doesn't work.
And then it has to be redone and then they are super-happy with the finished product because it turned out so good.
BUT they didn't got their directly!
(Sure windows may be one of those things which become more of a trouble in general in that show.. I still have a feel that maybe the people from UK built things quicker.)
I've also seen it where a team from the US came to build a house and they had brought a nail gun with them. It was just that the others didn't had any nail guns so now the build took a lot longer because people in Europe / in the UK nailed by hand.
And then you wonder why the US is the richer country with the higher salaries and is more competitive ..
(I'm from Sweden but I totally enjoy the English TV programs which is available here (which also is Sweden.))
If we're doing something important, like killing Hitler, or trying to beat the Commies to the moon, federal procurement can be remarkably efficient. Clear goals, and the stated willingness to accept some waste as long as the job's done, can do that.
Unfortunately since about the mid-1970s (Watergate you say?!), approximately zero "waste," of any kind is tolerated on any federal project, as this is "profiteering" or "wasting the people's money," so a lot of contractor time is spent on compliance. This makes the process incredibly loss-averse, and probably too risk-averse to actually accomplish anything.
The reality is that Elon Musk is able to do a good job, because he can destroy two or three recovery barges in a row and he doesn't have to explain it to anybody but his accountant. If the SLS had only one slip-up like that there would be a bloodbath of firings, senate hearings, press conferences with the President, and maybe the entire program might be scrubbed. Back in the late 50s NASA screwed up these kinds of operations all the time, but the American people tolerated it because of the Cold War. Nowadays the budget is so tight and public accountability is so fierce that frigging welding assembly subcontractors are apparently front-page news. We probably built and destroyed five facilities on the scale of this thing during Apollo and nobody batted an eye at the expense.
Don't blame me, I voted for Baltar.
Maybe the layed off people from MakerBot went to work for ESAB
http://www.engadget.com/2015/04/17/makerbot-lays-off-20-percent-of-its-employees/
Actually, that's only sort-of true. I started with MCC interim release, but couldn't get it to work properly. So then I spent a few days downloading SLS and it worked just fine - well, as good as you could expect with only 4MB of ram. But I didn't notice any alignment issues, and I wasn't instructed to reinforce the floor so I didn't. I had problems with overheating during compilation though, which I fixed by a powerful floor fan pointed at the air intake of the PC. I later fixed this more gracefully with a home-made triple-sized heat sink. Maybe that's what NASA should do, build a giant heat sink onto it.
That CAN be done under certain conditions, but unfortunately the political discourse of the time makes that politically expensive. For example, vice president Al Gore gave an award to one particular company and presented them as a case study of efficient and effective government contracting. He was right, they did a good job.
A few years later, when the Bush administration needed to have infrastructure rebuilt in Iraq, they turned to the same company. Since they were known to be good and they were one of only two or three companies who could quickly accomplish projects of that type, they got an efficient deal - here's what needs to be done, and here's what we'll pay, now get started. (As opposed to 4 1/2 years for just the bid process). For the next ten years, those who voted for Gore vilified Bush for hiring the company Gore presented an example of excellence, Halliburton. It doesn't matter how good and how efficient they are, people will vilify you if you don't waste half the money on a thousands of pages of bid documents over several years, followed by tens of thousands of pages of oversight and compliance.
Just work with companies with a good record. I worked at a place that had a policy to hire minority and woman owned businesses. Some were OK, others couldn't perform to the needed standards.
Asked why ESAB did not reinforce the foundation as it was supposed to, May said only it was a result of "a miscommunication between two [Boeing] subcontractors and ESAB."
Bullshit. The reason is they thought they could get away with it.
Musk hasn't destroyed *ANY* recovery barges, much less "two or three...in a row". In both failed landings, the barge suffered minimal (mostly cosmetic) damage.
Most of the rest of what you posted is similarly accurate.
I'm not saying you're wrong, but Cheney's ties to Halliburton make a difference. Even as an adult understands the reason why a stove element glows red, anyone experienced in politics knows there needs to be a bidding process that will stand up to an ethics audit if you're thinking of giving a contract to an outfit that used to employ you in a management position. If they didn't have time for the bidding, one of the other two or three companies should have been picked.
Halliburton wasn't vilified because of the lack of paperwork. They were vilified as a part of the whole botched up business of rebuilding a country. They were also an integral part of the Pentagon ideal of subcontracting war waging and the support operations.
What they build really well though was the military-diplomatic complex a.k.a the embassy, which had the purpose to defend the occupiers from the angry natives. Tells a lot doesn't it?
The reality is that Elon Musk is able to do a good job, because he can destroy two or three recovery barges in a row
The biggest reason is that he really wants to do a good job, because it's his own money and reputation.
If a contractor overruns his budget, and the result is that he gets a bigger budget, where's the motivation to do a good job ?
The final inspection yes, the in process QC not so much.
That's called a fixed price contract. There are millions of them on the GSA Schedule.
"The net effect on SLS’s development, both in terms of cost and possible delays"
Sounds like the taxpayers are the ones who are going to get to pay for this "miscommunication" (see attempted fraud). Ah, "cost plus" contracts, you never cease to amaze. I hope someone waves this, that A-3 test stand debacle and all of the other "miscommunications" in the faces of all of the people trying to get money diverted from the CCDev program, a contract based program, to SLS.
And therein lies the problem - What needs to happen is give private company the goal X with cost X, and not give a shit how it gets accomplished.
That is not always the best approach, since they may still cut corners. The real solution, IMO, is for the original customer to do acceptance testing and ensure the contract has penalties for failures to meet requirements. Acceptance testing should be done in-house or a separate sub-contractor, though where skills exist the former would be better.
Jumpstart the tartan drive.
It is baffling how everyone at NASA, Boeing, and ESAB could have forgotten to do the reinforcing
No what's baffling is that the metric system is still not in wide use in the USA.
That's baffling, a bunch of engineers screwing up is par for the course.
"If any question why we died, Tell them because our fathers lied."
Maybe that’s a way of generating profit when you have a government contract, make mistakes.
"If any question why we died, Tell them because our fathers lied."
"and the result is that he gets a bigger budget"
Not always true. Sometimes, he loses the contract and it goes to someone else. Sometimes, he loses the ability to get awarded future contracts. Sometimes, he gets sued but he government to recover the money. It totally depends on the cause.
If a contractor overruns his budget because of unforeseen technical difficulties, that is one thing. They usually are on the cutting edge, so predicting that sort of thing is difficult. If they overrun their budget due to incompetence, that is entirely different.
while(1) attack(People.Sandy);
If the SLS had only one slip-up like that there would be a bloodbath of firings, senate hearings, press conferences with the President, and maybe the entire program might be scrubbed.
I don't get your point. Isn't *THIS* a slip-up that is much worse than the SpaceX designed-to-be-expendable barges?
Violence is the last refuge of the incompetent. Polar Scope Align for iOS
When a sub-contractor screws up, Apple has to pay.
Of course news about a fake are Fake News.
Jet Line robotic welders never had this problem, think ISS. Is this another case of Congress throwing scrap meat to world market vendors and H1B hacks? Only to pay more for it later?
It's called, "not my job." Guaranteed to cause a project to fail, every time.
Not always true. Sometimes, he loses the contract and it goes to someone else
That's a matter of skillful management. Make sure that the client is already invested too much, and that the overrun would cost less than finding someone else for the contract. Also, make sure that the original contract is written in a way that the budget is not guaranteed, and that you can blame the overrun on the client (poor specification). And finally, a nice bribe is always helpful.
Of course, when you write specification "X", there are usually some mistakes. A contractor may spot the mistakes, but follow the letter of the specification anyway. When everything is finished, and you discover that "X" was wrong, you get an additional contract to fix things.
Then, there's a tendency towards analysis paralysis at NASA. Sometimes, it's better to just build it and try it, particularly with complex interacting systems where the analytical tools aren't available.
NASA is the kind of place that if you were installing a FM radio in your lab to listen to music while you work, they would do a complete spectrum survey, consider 7 different kinds of antennas, develop theoretical models for the antennas and the building on which the antenna will be mounted, considering the location of the FM transmitters, the propagation paths from those transmitters to the antennas, calculate link margins for all possible weather conditions, and then engage in some foundational research to determine the effect of the music on work quality, and whether intermittent weather related fades in the signal might have an effect.
At SpaceX, they'd just put a wire out the window, and if it worked ok, great, call it done..
So, if the day comes when the music dies, at SpaceX, they won't necessarily know what caused it, but hey, we can try something new. But at NASA, they'll be able to support a full congressional inquiry and generate a 500 slide report with graphs, arrows, the analysis from 32 PhDs, 14 astronauts, and 17 financial and schedule managers and be able to tell you precisely, to the hundredth of a cent and work minute of schedule, what could have been done instead.
NASA is incredibly risk averse, so there tends to be a "let's consider every possible thing that could go wrong, and develop an analysis to evaluate the probability and consequence so we can plot them on our 5x5" and then, combined with such things as Earned Value Reporting, and an incredibly stultifying document review process that focuses on tiny text changes, rather than whether it makes sense (in each RID (that's a document comment), please specify the exact "from" and "to" wording requested, and provide an estimate of the cost and/or risk changes as a result of this change) So you get a lot of "wordsmithing" rather than "are we building the right thing"
There's a fair number of over 50s at SpaceX. Yeah, some of them can't take the "startup-like" intensity. SpaceX is a "here is our ONE goal: you are working to that goal". SpaceX doesn't send people to conferences. They don't publish papers or reports. That doesn't help you get to the goal. You'll not see SpaceX trumpeting "spin-offs" like Tang. Someone else can do that, at SpaceX we're going to Mars, if you're going to Mars, you're doing the right thing, if you're doing something else, you're doing the wrong thing, and you might find it happier working somewhere else.
I dunno. If someone had been micromanaging it, maybe they would have remembered to reinforce the floor.
Even if NASA and ESAB had a "miscommunication" (I suspect an unresolved contract issue, which both sides thought the other has accepted responsibility for owning the floor contracting), what should have happened is that the ESAB equipment people, before starting work on the installation should have inspected the floor work they mandated to make sure it was done correctly. If this happened at all, you'd assume someone who notice that the floor has not been recently rebuilt AT ALL and would stop work until that got done. If you say your equipment needs some part of the environment to be a certain way before you can install, presumably you don't do it until it meets spec. So, no matter who else is to blame, ESAB is negligent in proceeding with work if the floor had not been brought in line with requirements.
An alternate, plausible chain of events is that NASA originally, disagreed with ESAB and felt the floor fix was unnecessary in the first place and told them if they wanted to do it, NASA was not going to pay for that. ESAB does a risk assessment, decides there's a danger but it likely will work and goes ahead. Install fails and during resolution, NASA makes under-the-table concessions to make ESAB whole financially if they admit it's their screw-up. This perfectly reflects the difference between govt and corporate fears. NASA fears looking stupid and is probably willing to pay money to avoid that. ESAB is more worried about losing money and can always subtly imply privately to other future customers that it was NASA that screwed up.
Maybe we've inadvertently turned them into compliance engineers instead of rocket engineers. If they can make money by doing paperwork instead of science...
Sometimes, he loses the contract and it goes to someone else. Sometimes, he loses the ability to get awarded future contracts
2014 EU Procurement Directives now allow contracting authorities to take past failures into account when evaluating tenders.
At last!
https://www.gov.uk/government/...
This is why we can never have nice things under capitalism. I'm sure the contractor noticed this early in the construction process, and they took the gamble that maybe nobody would notice and saved some dollars. They got caught, and so it cost them... But if you think this is an isolated incident, you're sadly delusional. Pretty much every company out there pulls stunts like this, and most of them don't get caught. That's why they do it. Worship the all mighty ROI.
Given my knowledge of Boeing, the problem isn't with "quality"; the problem is with bad management -- and a culture of failure to admit that management can do no wrong. This can be easily exposed: Take a look at the jobs at Boeing. Look for a software tester / QA position. You will be lucky to find ONE. The jobs you do find are not test/qa; but rather development that can test their own code to the specifications written. And here is where QA comes in: what it there is a problem with the spec? And there is testing: How much testing should be done.
I remember / know of at least two (somewhat recent) incidents with Boeing aircraft that (for anyone that knows software) resulted in a crash of the aircraft -- that can be directly traced to this culture of management can do no wrong and developers test their own code.