Slashdot Mirror
Incorrectly Built SLS Welding Machine To Be Rebuilt
schwit1 writes A giant welding machine, built for NASA's multi-billion dollar Space Launch System (SLS), has to be taken apart and rebuilt because the contractor failed to reinforce the floor, as required, prior to construction: "Sweden's ESAB Welding & Cutting, which has its North American headquarters in Florence, South Carolina, built the the roughly 50-meter tall Vertical Assembly Center as a subcontractor to SLS contractor Boeing at NASA's Michoud Assembly Facility in New Orleans.
ESAB was supposed to reinforce Michoud's floor before installing the welding tool, but did not, NASA SLS Program Manager Todd May told SpaceNews after an April 15 panel session during the 31st Space Symposium here. As a result, the enormous machine leaned ever so slightly, cocking the rails that guide massive rings used to lift parts of the 8.4-meter-diameter SLS stages The rings wound up 0.06 degrees out of alignment, which may not sound like much, "but when you're talking about something that's 217 feet [66.14 meters] tall, that adds up," May said.
Asked why ESAB did not reinforce the foundation as it was supposed to, May said only it was a result of "a miscommunication between two [Boeing] subcontractors and ESAB."
It is baffling how everyone at NASA, Boeing, and ESAB could have forgotten to do the reinforcing, even though it was specified in the contract. It also suggests that the quality control in the SLS rocket program has some serious problems.
ESAB was supposed to reinforce Michoud's floor before installing the welding tool, but did not, NASA SLS Program Manager Todd May told SpaceNews after an April 15 panel session during the 31st Space Symposium here. As a result, the enormous machine leaned ever so slightly, cocking the rails that guide massive rings used to lift parts of the 8.4-meter-diameter SLS stages The rings wound up 0.06 degrees out of alignment, which may not sound like much, "but when you're talking about something that's 217 feet [66.14 meters] tall, that adds up," May said.
Asked why ESAB did not reinforce the foundation as it was supposed to, May said only it was a result of "a miscommunication between two [Boeing] subcontractors and ESAB."
It is baffling how everyone at NASA, Boeing, and ESAB could have forgotten to do the reinforcing, even though it was specified in the contract. It also suggests that the quality control in the SLS rocket program has some serious problems.
is a penny burned
“He’s not deformed, he’s just drunk!”
The contractors that NASA has used have squandered and bilked NASA for all they are worth - If you give the goal of what the SLS does to SpaceX - (and not micromanage it) Elon Musk will have it completed in a much shorter amount of time.
... is ten dollars on the shop floor and a hundred dollars in the field.
Looks like they tried to save eleven dollars and got caught out!
More complicated than this NASA shit
Job security for the government and libtards who voted them in
The more screw ups the SLS has, the more likely it will get cancelled. The SLS has not had many visible screw ups so far. Maybe its to the credit of NASA. Maybe its for sticking with what you know. Maybe NASA has not gotten to the point where screw ups occur. Maybe the screw ups are not public yet. Cancellation might be for the better. NASA has not funded any new payloads for the SLS yet...
ESAB just announced they were closing their Florence headquarters in the US and sending the jobs to Mexico. I guess they want to save money on all fronts.
The article does not mention where the cost of this error is going to fall. This seems like an important detail. On a sufficently complex project, one of the bevy of subcontractors fucking something up isn't a huge surprise; but I would be very, very, disappointed if NASA wasn't able to contract sufficiently vigorously to make the vendor eat the cost of delivering the goods as specified, rather than paying them for their effort no matter how well or badly they do.
Can't trust them with a dime.
Its nearly 3 inces. What was the precision of the Saturn 5?
"and not micromanage it". That's the rub. The micromanaging, the reporting and compliance costs, can be over 50% of the cost for some federal contracts, but most of the time that's required by thousands upon thousands of pages of federal law. When you have a comoany that knows how to do a certain thing , aka one of those evil corporations, getting hired by the federal government, some people want to do a lot of paperwork and stuff to keep track of what's going on, and other people go crazy with it. The organization I work for used to do a lot of federal contracts. We quit and now just do state contracts for states that are reasonable.
Still other people added a bunch of requirements for federal contracting that aren't really relevant to the project. For example, how many black women work for each of your major suppliers? How much do your interns make? Are all of the web pages and documentation you've ever made fully accessible to people who are both blind and deaf?
We quit dealing with the feds and certain states because it's just not worth it. It would cost SPACEX five times as much to build a federally-contracted rocket than it costs to build their own.
I'm sure all the paperwork was in order, and they had many people sign off and assure the configuration prior to building the thing.
http://en.wikipedia.org/wiki/NOAA-19
http://www.spaceflightnow.com/news/n0410/04noaanreport/
Whether anyone actually went out and looked at the floor is another story.
I'd say if they noticed something was six hundredths of a degree off, their quality control is just fine.
Age old story of outsourcing - you still need to retain enough people to watch the contractors so they can't cut corners on the expensive bits.
One blatant example I saw was with non destructive testing of welds in high pressure pipework leading to portions of a turbine in a coal fired power station. At those welds it was done by spraying on thing white paint, using a magnet and spraying on a fluid with suspended magnetic "dust" that would collect wherever defects disrupted the magnetic field. Access was a bit tight so the contractors tested the top of the pipes and they ran the magnet around the bottom of the pipe without looking at it so that some scratches would be left to show that the magnet had been used. The lazy pricks were caught doing that so we had to send someone along as an observer and make them do a couple of weeks worth of work over again, because with their scratch trick we had no way of knowing is any inspection had actually been done or not.
So MBA types - that person standing off to the side not doing anything during a concrete pour may be there solely to reduce fuckups due to dishonest contractors.
we talk of pennies gained not burned,
your on the wrong page.
they made more pennies this way.
I hate watching Grand Designs and then there's someone who has ordered three glass insulation windows from Sweden (most likely the case, it may just have happened like two times or whatever but still!) and then it either takes forever or there's wrong measurements / doesn't work.
And then it has to be redone and then they are super-happy with the finished product because it turned out so good.
BUT they didn't got their directly!
(Sure windows may be one of those things which become more of a trouble in general in that show.. I still have a feel that maybe the people from UK built things quicker.)
I've also seen it where a team from the US came to build a house and they had brought a nail gun with them. It was just that the others didn't had any nail guns so now the build took a lot longer because people in Europe / in the UK nailed by hand.
And then you wonder why the US is the richer country with the higher salaries and is more competitive ..
(I'm from Sweden but I totally enjoy the English TV programs which is available here (which also is Sweden.))
I did machine controls at that plant, and it was amazing the number of people there that were proud they couldn't read.
Sounds more like a union shop. Unions are notoriously allied with the Democrat party by the way.
If we're doing something important, like killing Hitler, or trying to beat the Commies to the moon, federal procurement can be remarkably efficient. Clear goals, and the stated willingness to accept some waste as long as the job's done, can do that.
Unfortunately since about the mid-1970s (Watergate you say?!), approximately zero "waste," of any kind is tolerated on any federal project, as this is "profiteering" or "wasting the people's money," so a lot of contractor time is spent on compliance. This makes the process incredibly loss-averse, and probably too risk-averse to actually accomplish anything.
The reality is that Elon Musk is able to do a good job, because he can destroy two or three recovery barges in a row and he doesn't have to explain it to anybody but his accountant. If the SLS had only one slip-up like that there would be a bloodbath of firings, senate hearings, press conferences with the President, and maybe the entire program might be scrubbed. Back in the late 50s NASA screwed up these kinds of operations all the time, but the American people tolerated it because of the Cold War. Nowadays the budget is so tight and public accountability is so fierce that frigging welding assembly subcontractors are apparently front-page news. We probably built and destroyed five facilities on the scale of this thing during Apollo and nobody batted an eye at the expense.
Don't blame me, I voted for Baltar.
You are correct. I live in Allendale not far from there, and only 21% of the people here voted for Romney in the last presidential election. I don't think a single Republican has been elected since Reconstruction when the Republicans wouldn't let whites vote.
What is this nonsense? Welding? Reinforced floors?
I WAS ASSURED THERE WAS A 3D PRINTING REVOLUTION THAT CHANGED THE GAME
FOREVER!!!!!
Now I'm reading about faggoty welding? WELDING? What is this? The 19th century??
FAGS!!!!
Now where's my Makerbot?
The SLS should be scrapped anyway, the only real reason it exists (at least in its current form) is to keep a bunch of contractors in key congressional districts in business after the end of the shuttle program.
And therein lies the problem - What needs to happen is give private company the goal X with cost X, and not give a shit how it gets accomplished.
Actually, that's only sort-of true. I started with MCC interim release, but couldn't get it to work properly. So then I spent a few days downloading SLS and it worked just fine - well, as good as you could expect with only 4MB of ram. But I didn't notice any alignment issues, and I wasn't instructed to reinforce the floor so I didn't. I had problems with overheating during compilation though, which I fixed by a powerful floor fan pointed at the air intake of the PC. I later fixed this more gracefully with a home-made triple-sized heat sink. Maybe that's what NASA should do, build a giant heat sink onto it.
That CAN be done under certain conditions, but unfortunately the political discourse of the time makes that politically expensive. For example, vice president Al Gore gave an award to one particular company and presented them as a case study of efficient and effective government contracting. He was right, they did a good job.
A few years later, when the Bush administration needed to have infrastructure rebuilt in Iraq, they turned to the same company. Since they were known to be good and they were one of only two or three companies who could quickly accomplish projects of that type, they got an efficient deal - here's what needs to be done, and here's what we'll pay, now get started. (As opposed to 4 1/2 years for just the bid process). For the next ten years, those who voted for Gore vilified Bush for hiring the company Gore presented an example of excellence, Halliburton. It doesn't matter how good and how efficient they are, people will vilify you if you don't waste half the money on a thousands of pages of bid documents over several years, followed by tens of thousands of pages of oversight and compliance.
Just work with companies with a good record. I worked at a place that had a policy to hire minority and woman owned businesses. Some were OK, others couldn't perform to the needed standards.
Compliance on a cost plus project is profit to the contractor.
Asked why ESAB did not reinforce the foundation as it was supposed to, May said only it was a result of "a miscommunication between two [Boeing] subcontractors and ESAB."
Bullshit. The reason is they thought they could get away with it.
This man gets it, +1 if i cared about logging in
Musk hasn't destroyed *ANY* recovery barges, much less "two or three...in a row". In both failed landings, the barge suffered minimal (mostly cosmetic) damage.
Most of the rest of what you posted is similarly accurate.
I'm not saying you're wrong, but Cheney's ties to Halliburton make a difference. Even as an adult understands the reason why a stove element glows red, anyone experienced in politics knows there needs to be a bidding process that will stand up to an ethics audit if you're thinking of giving a contract to an outfit that used to employ you in a management position. If they didn't have time for the bidding, one of the other two or three companies should have been picked.
Halliburton wasn't vilified because of the lack of paperwork. They were vilified as a part of the whole botched up business of rebuilding a country. They were also an integral part of the Pentagon ideal of subcontracting war waging and the support operations.
What they build really well though was the military-diplomatic complex a.k.a the embassy, which had the purpose to defend the occupiers from the angry natives. Tells a lot doesn't it?
This sounds like someone has a bit of an axe to grind.
Mistakes happen in the construction industry, they can be fixed it just requires time.
But hey paint the whole industry with the broad brush labeling us as evil.
A few things come to mind:
1. If it was required by the contract, how many people knew about it?
2. Did anyone in management push to ignore the issue to complete their task?
blame someone else
There's no magic and probably no malice here. It is an old case of "Fast. Cheap. Good. Pick two"
Say it ain't so!
The reality is that Elon Musk is able to do a good job, because he can destroy two or three recovery barges in a row
The biggest reason is that he really wants to do a good job, because it's his own money and reputation.
If a contractor overruns his budget, and the result is that he gets a bigger budget, where's the motivation to do a good job ?
That's called a fixed price contract. There are millions of them on the GSA Schedule.
"The net effect on SLS’s development, both in terms of cost and possible delays"
Sounds like the taxpayers are the ones who are going to get to pay for this "miscommunication" (see attempted fraud). Ah, "cost plus" contracts, you never cease to amaze. I hope someone waves this, that A-3 test stand debacle and all of the other "miscommunications" in the faces of all of the people trying to get money diverted from the CCDev program, a contract based program, to SLS.
they made more pennies this way.
Who are "they"? The contractor that was required to reinforce the floor but did not? If the contract was written correctly, "they" burned pennies, they did not make more pennies.
An IKEA product obviously!
And therein lies the problem - What needs to happen is give private company the goal X with cost X, and not give a shit how it gets accomplished.
That is not always the best approach, since they may still cut corners. The real solution, IMO, is for the original customer to do acceptance testing and ensure the contract has penalties for failures to meet requirements. Acceptance testing should be done in-house or a separate sub-contractor, though where skills exist the former would be better.
Jumpstart the tartan drive.
It is baffling how everyone at NASA, Boeing, and ESAB could have forgotten to do the reinforcing
No what's baffling is that the metric system is still not in wide use in the USA.
That's baffling, a bunch of engineers screwing up is par for the course.
"If any question why we died, Tell them because our fathers lied."
Maybe that’s a way of generating profit when you have a government contract, make mistakes.
"If any question why we died, Tell them because our fathers lied."
"and the result is that he gets a bigger budget"
Not always true. Sometimes, he loses the contract and it goes to someone else. Sometimes, he loses the ability to get awarded future contracts. Sometimes, he gets sued but he government to recover the money. It totally depends on the cause.
If a contractor overruns his budget because of unforeseen technical difficulties, that is one thing. They usually are on the cutting edge, so predicting that sort of thing is difficult. If they overrun their budget due to incompetence, that is entirely different.
while(1) attack(People.Sandy);
You forgot the context of your statement: all was not on the board of directors of the appointed company, dick was and still is, part owner.
My dad works in construction and most of his job is about dealing with this kind of contractor issue. Now, if he had been on the project he would have insisted upon seeing the floor with his own eyes before the machine was put in place and this never would have happened (he is exceptionally good at his job). HOWEVER, this kind of communications nightmare between contractor and client does happen and given the amount of things that need to be communicated between the two it is inevitable that some things will be misinterpreted, or that the contractor will slip cost savings passed the client. This happens just as often in private industry. The difference is that NASA actually has to tell us about it, and furthermore that we actually care.
If the SLS had only one slip-up like that there would be a bloodbath of firings, senate hearings, press conferences with the President, and maybe the entire program might be scrubbed.
I don't get your point. Isn't *THIS* a slip-up that is much worse than the SpaceX designed-to-be-expendable barges?
Violence is the last refuge of the incompetent. Polar Scope Align for iOS
Gersh gurndy morn-dee burn-dee, burn-dee, floordee hardee
- Swedish Chef
When a sub-contractor screws up, Apple has to pay.
Of course news about a fake are Fake News.
Jet Line robotic welders never had this problem, think ISS. Is this another case of Congress throwing scrap meat to world market vendors and H1B hacks? Only to pay more for it later?
Not always true. Sometimes, he loses the contract and it goes to someone else
That's a matter of skillful management. Make sure that the client is already invested too much, and that the overrun would cost less than finding someone else for the contract. Also, make sure that the original contract is written in a way that the budget is not guaranteed, and that you can blame the overrun on the client (poor specification). And finally, a nice bribe is always helpful.
Of course, when you write specification "X", there are usually some mistakes. A contractor may spot the mistakes, but follow the letter of the specification anyway. When everything is finished, and you discover that "X" was wrong, you get an additional contract to fix things.
Then, there's a tendency towards analysis paralysis at NASA. Sometimes, it's better to just build it and try it, particularly with complex interacting systems where the analytical tools aren't available.
NASA is the kind of place that if you were installing a FM radio in your lab to listen to music while you work, they would do a complete spectrum survey, consider 7 different kinds of antennas, develop theoretical models for the antennas and the building on which the antenna will be mounted, considering the location of the FM transmitters, the propagation paths from those transmitters to the antennas, calculate link margins for all possible weather conditions, and then engage in some foundational research to determine the effect of the music on work quality, and whether intermittent weather related fades in the signal might have an effect.
At SpaceX, they'd just put a wire out the window, and if it worked ok, great, call it done..
So, if the day comes when the music dies, at SpaceX, they won't necessarily know what caused it, but hey, we can try something new. But at NASA, they'll be able to support a full congressional inquiry and generate a 500 slide report with graphs, arrows, the analysis from 32 PhDs, 14 astronauts, and 17 financial and schedule managers and be able to tell you precisely, to the hundredth of a cent and work minute of schedule, what could have been done instead.
NASA is incredibly risk averse, so there tends to be a "let's consider every possible thing that could go wrong, and develop an analysis to evaluate the probability and consequence so we can plot them on our 5x5" and then, combined with such things as Earned Value Reporting, and an incredibly stultifying document review process that focuses on tiny text changes, rather than whether it makes sense (in each RID (that's a document comment), please specify the exact "from" and "to" wording requested, and provide an estimate of the cost and/or risk changes as a result of this change) So you get a lot of "wordsmithing" rather than "are we building the right thing"
There's a fair number of over 50s at SpaceX. Yeah, some of them can't take the "startup-like" intensity. SpaceX is a "here is our ONE goal: you are working to that goal". SpaceX doesn't send people to conferences. They don't publish papers or reports. That doesn't help you get to the goal. You'll not see SpaceX trumpeting "spin-offs" like Tang. Someone else can do that, at SpaceX we're going to Mars, if you're going to Mars, you're doing the right thing, if you're doing something else, you're doing the wrong thing, and you might find it happier working somewhere else.
I dunno. If someone had been micromanaging it, maybe they would have remembered to reinforce the floor.
Even if NASA and ESAB had a "miscommunication" (I suspect an unresolved contract issue, which both sides thought the other has accepted responsibility for owning the floor contracting), what should have happened is that the ESAB equipment people, before starting work on the installation should have inspected the floor work they mandated to make sure it was done correctly. If this happened at all, you'd assume someone who notice that the floor has not been recently rebuilt AT ALL and would stop work until that got done. If you say your equipment needs some part of the environment to be a certain way before you can install, presumably you don't do it until it meets spec. So, no matter who else is to blame, ESAB is negligent in proceeding with work if the floor had not been brought in line with requirements.
An alternate, plausible chain of events is that NASA originally, disagreed with ESAB and felt the floor fix was unnecessary in the first place and told them if they wanted to do it, NASA was not going to pay for that. ESAB does a risk assessment, decides there's a danger but it likely will work and goes ahead. Install fails and during resolution, NASA makes under-the-table concessions to make ESAB whole financially if they admit it's their screw-up. This perfectly reflects the difference between govt and corporate fears. NASA fears looking stupid and is probably willing to pay money to avoid that. ESAB is more worried about losing money and can always subtly imply privately to other future customers that it was NASA that screwed up.
Ben Rich, the second Skunk Works director after Kelly Johnson, said in Skunk Works:... that around the end of the F-117 project, that Federal regulations had become so oppressive that they wouldn't be able to duplicate their prior successes. The U-2 was contracted in 1954 and flew in 1955, the SR-71 was contracted at the end of 1962 and flew in 1964.
By way of contrast, after burning through $1 billion, the X-33 never flew and its engine was never built.
The contractor may spot the mistake, but be forced to follow the letter of the specification anyway - because to change the spec would require time (pushes delivery date to the right), money (increases budget), and an admission that the original spec was incorrect (mud on the face of the original contracting officer).
No one in the government is willing to force their boss to look bad, as well as take responsibility for delaying delivery and increasing cost, when doing it wrong costs them absolutely nothing.
Maybe we've inadvertently turned them into compliance engineers instead of rocket engineers. If they can make money by doing paperwork instead of science...
Sometimes, he loses the contract and it goes to someone else. Sometimes, he loses the ability to get awarded future contracts
2014 EU Procurement Directives now allow contracting authorities to take past failures into account when evaluating tenders.
At last!
https://www.gov.uk/government/...
This is why we can never have nice things under capitalism. I'm sure the contractor noticed this early in the construction process, and they took the gamble that maybe nobody would notice and saved some dollars. They got caught, and so it cost them... But if you think this is an isolated incident, you're sadly delusional. Pretty much every company out there pulls stunts like this, and most of them don't get caught. That's why they do it. Worship the all mighty ROI.
yes, sadly, it is. It also generates more work as the easiest way to get it fixed is to mod the contract.
This was constructed within a government facility (Michoud) where the Saturn V moonrocket tanks and shuttle external tanks were made - so there is ZERO possibility that NASA management did not know about this (They were watching the construction, photographing it, posting updates about it on the web, etc).
Boeing, the chosen SLS contractor, has built some of the world's largest aerospace fabrication facilities for the production of its aircraft - so there is ZERO possibility that Boeing management did not know how to do this themselves, or properly oversee a subcontractor doing it.
There IS a plausible answer however that involves neither extreme gullibility nor an aluminum foil hat: The Obama administration HATES the SLS and has tried to kill it many times in many ways but they have been thwarted by the US Senate who ordered it built. In every year of the project, they have tried to reduce the SLS budget and found a nearly unending list of reasons why its schedule must slip. This administration also has built a long trackrecord of extreme passive-aggressive behavior towards its opponents in congress on EVERYTHING. (Obamacare rammed through on an invalid budget procedure called "Reconcilliation", Obamacare repeatedly altered by re-interpreting words rather than ammendments passed by congress, Immigration by simply ordering immigration to let everybody in, executive appointments installed without required senate approval, major policy changes on Iran and Cuba done without congress on various executive branch edicts, and so-on). It seems more that slightly possible that they found a way to slip the schedule further (in the hope congress will abandon SLS) by ALLOWING the tooling to be mis-built, and that Boeing did not want the political heat in congress for participating in that so they farmed-it-out to a foreign firm that is owned by a US ally (who is therefore effectively shielded from any real blowback).
If you support SLS, there is little you can do about this; The members of congress appear to have become afraid of being called "racists" if they actually try to block Obama on big important things, so they have stopped offering any real resistance. THIS issue is not even "big" in the mind of the public, so it's absolutely NOT something they will draw a line in the sand on, and it will suit their desire to avoid a fight to simply ignore it. The law congress wrote in 2010 required SLS to make its first flight in fiscal 2017 (NOT unreasonable for a rocket using proven tech and proven engines) and the President has already tested them on this, first by pushing it to December 2017 (the 1st quarter of fiscal 2018 and "re-interpreting" the law to say the flight needs to be in CALENDAR 2017), and then by dropping the fiscal/calendar year argument and pushing the launch into "late 2018" - without getting any serious resistance from congress.
If you oppose SLS then this will look like yet another "clever" tactic to overcome congress.
Hire a bunch of ignorant scabs from Sweden, you get crap. Same with H-1b crap, it's all garbage. Could these morons read the specs?
He said Musk *can* destroy two or three barges, not that he *did* destroy them.
CAPTCHA: swimsuit
Given my knowledge of Boeing, the problem isn't with "quality"; the problem is with bad management -- and a culture of failure to admit that management can do no wrong. This can be easily exposed: Take a look at the jobs at Boeing. Look for a software tester / QA position. You will be lucky to find ONE. The jobs you do find are not test/qa; but rather development that can test their own code to the specifications written. And here is where QA comes in: what it there is a problem with the spec? And there is testing: How much testing should be done.
I remember / know of at least two (somewhat recent) incidents with Boeing aircraft that (for anyone that knows software) resulted in a crash of the aircraft -- that can be directly traced to this culture of management can do no wrong and developers test their own code.
TYpically, reinforcing the floor means tearing it out, digging a big hole, and filling it back in. it takes months and creates a big mess.
thats kind of hard to miss when its not being done.no one questioned why there wasnt a big hole at any time?