Slashdot Mirror
Excessive Modularity Hindered Development of the 787
TAGmclaren writes "The Harvard Business Review is running a fascinating article exploring the issues facing Boeing's Dreamliner. Rather than simply blaming outsourcing, as much of the commentary has been focused on, the article delves into the benefits of integration and how being integrated when developing a new product gives engineers more degrees of freedom. From the article: 'Historically, Boeing understood that, and had worked with its subcontractors on that basis. If it was going to rely on them, it would provide them with detailed blueprints of the parts that were required — after Boeing had already created them. That, in turn, meant that Boeing had to design all the relevant pieces of the puzzle itself, first. But with the 787, it appears that Boeing tried a very different approach: rather than having the puzzle solved and asking the suppliers to provide a defined puzzle piece, they asked suppliers to create their own blueprints for parts. The puzzle hadn't been properly solved when Boeing asked suppliers for the pieces. It should come as little surprise then, that as the components came back from far-flung suppliers, for the first plane ever made of composite materials... those parts didn't all fit together. Time and cost blew out accordingly. It's easy to blame the outsourcing. But, in this instance, it wasn't so much the outsourcing, as it was the decision to modularize a complicated problem too soon.'"
So Boeing told the contractors what they needed to build, but didn't give them hard specifications? What the hell? Two things:
Boeing needs to have their collective asses kicked for doing it this way, and:
The subcontractors should never have agreed to the work without specs first.
The first one is probably the result of Boeing not wanting to spend the engineering dollars to develop the blueprints, and the second is due to the enormous amounts of money involved in making the parts.
Now that I know this, you'll never catch me on one of those abominations. What the hell was Boeing thinking?
Never underestimate the power of stupid people in large groups.
I suspect that with the mindset of a government contractor they told a bunch of different places sort of what they wanted, but not where it had to be or what it was going to plug into. Measurements, plugs, protocol standards, bolt holes, shape - they all matter - and "on a plane" doesn't answer most of the questions.
The preceding post was not a Slashvertisement.
Systems design in engineering basically involves drawing a box around a bunch of parts and saying "this is a system". The interfaces after that are hopefulyl clean -- good systems design does that, but implicit in the choice of a system breakdown is efficiency loss. I might not, for example, think about the fact that the giant engine at the heart of my car could also run heating. There's this long term conflict in engineering between the need to abstract, which enables all forms of delegation, including outsourcing, subcontracting and even building teams, and the loss of efficiency. Good engineers learn things at an almost inexpressible level,developing jargons for the systems under their purview -- in the case of Boeing, there was literally one guy who was their expert on cabling. If you wanted to submit a drawing change, he could envision the change in the cabling of the plane and whether the change was physically possible. That's always been the bane of system abstraction - you find these things that have to cross systems and, if you don't recognize them early enough, they come back to bite you in all sorts of creative ways. Kelly Johnson was a big believer in this. His rules for skunkworks explicitly required that engineers had to be within a specific number of feet of the shop floor -- that way they weren't too divorced from the reality of the products they were making. You see this in the design of a lot of the early computer systems as well, parts bolted together in weird ways before we started developing this high-level view of what systems actually made up a computer.
Obviously, Boeing should simply have specified that all the contractors deliver components that accept and output plaintext, and then used pipes and awk to cobble the pieces together into a working system! What could possibly go wrong?
Boeing didn't want to hire all the engineers needed to design the 787. So when they outsourced these subsystems they also counted on their suppliers to do the engineering of these subsystems.
The problem is that engineers are not fungible. Boeing didn't appreciate this, any more than the software industry did when it started outsourcing.
An aerospace structural frame engineer is not the same thing as a marine structure engineer. There are huge differences in the body of experience despite the fact that they both use the same tools.
This was the primary cause of the delays Boeing had. It will continue to be a problem for anyone who tries this sort of outsourcing.
In it, he termed the architecture of the GS Yuasa battery packs supplied to Boeing "inherent unsafe," and predicted more fires from the same causes due to its design.
Specifically, Musk criticized the use of large-format lithium-ion cells "without enough space between them to isolate against the cell-to-cell thermal domino effect."
He also noted that when thermal runaway occurs in the larger cells, more energy is released by the single cell than comes from a small-format "commodity" cell, of the type used by the thousands in Tesla battery packs.
And he went on to highlight what he viewed as the dangers of batteries using those large-format cells, saying they have a "fundamental safety issue" because it's harder to keep the internal temperature of a large-format cell consistent from the center to the edges.
Not surprisingly, Mike Sinnett--Boeing's chief engineer for the 787 project--counters that the company designed the pack to cope with not only a single cell failure but to contain runaway thermal events as well."
http://www.greencarreports.com/news/1082007_tesla-ceo-musk-boeing-787-batteries-inherently-unsafe
There was a short article on the Dreamliner in the latest New Yorker magazine REQUIEM FOR A DREAMLINER? . Quote Surowiecki :The Dreamliner was supposed to become famous for its revolutionary design. Instead, it’s become an object lesson in how not to build an airplane.
To understand why, you need to go back to 1997, when Boeing merged with McDonnell Douglas. Technically, Boeing bought McDonnell Douglas. But, as Richard Aboulafia, a noted industry analyst with the Teal Group, told me, “McDonnell Douglas in effect acquired Boeing with Boeing’s money.” McDonnell Douglas executives became key players in the new company, and the McDonnell Douglas culture, averse to risk and obsessed with cost-cutting, weakened Boeing’s historical commitment to making big investments in new products. Aboulafia says, “After the merger, there was a real battle over the future of the company, between the engineers and the finance and sales guys.” The nerds may have been running the show in Silicon Valley, but at Boeing they were increasingly marginalized by the bean counters.
Read more: http://www.newyorker.com/talk/financial/2013/02/04/130204ta_talk_surowiecki#ixzz2JTGx7SPc
TLDR scalability issue due to cubed vs square law scaling.
High temperature can blow up some Li cells. The total thermal energy scales with volume, but the surface heat can escape from scales with surface area. So its hard (although not impossible) for one cell in a laptop to blow up the adjacent cells. But if you make the individual cells big enough you can get a chain reaction going.
This is a meta issue anyway. There are battery techs not susceptible to chain reactions, and not susceptible to occasionally blowing up an individual cell anyway.
Lets say you decide to make mousepads out of fissionable plutonium. Boss reports that they're getting bad PR because of mfgr variations or mishandling sometimes the plutonium mousepad blows up, although you tried to design it for a very unfavorable geometry. Well you can argue geometry and mfgr tolerances all day, but the mistake was making the mousepad out of plutonium, not making it "wrong".
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
I think the article (and the person talking) meant that they hoped it was a battery problem because they would then have isolated the problem to a single component, which is much easier to fix.
If the problem is systemic, then it can be orders of magnitude harder to fix. For example, is it because the components, whilst each individually OK, behave in strange ways when combined in a certain way. Is the issue an emergent property of the whole system or only of part of the system? And which part. Is the part that appears to fail the actjal part that fails, or has something else failed (and affected another part badly)?
A completely made up example would be that a voltage regulator fails, and there is a voltage spike somewhere causing another part to melt. How do you know what caused that part to melt, particularly if you weren't monitoring the voltage. (Obviously stupid example.)
Almost 40% of a Boeing 777 by weight is foreign sourced (not including engines) so they didn't have to build it n pieces to include foreign suppliers - aside from that, the point of the article is that Boeing also gave the job of detailed design definition to the outsourced suppliers, and that is where the issue comes in.
Aircraft have been built in pieces for decades before the 787, for example all Airbus aircraft since the A320 in the mid 1980s have been built as prefabricated sections and joined on the FAL in exactly the same way as that intended for the 787. Airbus have only had one major issue with this approach, the software issues in CATIA version mismatches that caused the A380 fuck up - it worked perfectly for every aircraft before.
When it comes to mechanical parts, geometric dimensioning and tolerancing is a solved problem. When it comes to electrical interoperability, one'd think that's a solved problem as well.
"Solved problem"? HAAHAHAHHHAAHHAAAAA....
You don't manufacture things for a living do you? I run a company that makes wire harnesses. We're a contract manufacturer - we don't design things, we just take prints and build what is on the prints. I can count on my fingers on one hand the number of prints we have gotten from customers which were correct and sufficiently detailed such that the product could be built without asking any questions. There pretty much always are critical details left out of the prints. About 2/3 of the prints we see have incompatible parts specified. About half are missing at least one important dimension such as length. About 10% have missing parts and about 25% have incompatible parts. About 20% specify needlessly expensive parts like gold plated terminals that cost more but provide no actual performance benefit. Most of them leave off at least one critical tolerance. I've even seen drawings with dimension in inches and tolerances in metric.
Why does this happen? For the most part because an alarming number of engineers doing the drawings aren't actually very good at their job. Some of them are just plain lazy. The electrical engineers usually can specify a wire schematic but often have no idea whether something can actually be built or know much about industry standards. The more mechanical engineers (yes mechanical engineers can and do design circuits) tend to create bad designs and specify the wrong parts because they don't know any better. Sometimes they are trying to do a good job but they don't bother to consult manufacturing during the design process and they come up with a stupid design or something that is impossible to build.
I have run into some good engineers but they are the exception.
Obviously a large project has to have an overarching design and direction but a great example of a failed top down aviation design would be the Space Shuttle. They designed many of the larger systems in oddly specific waves of a wand and then left it to engineers to actually invent them. A really great example of this failure were the cryopumps for the liquid hydrogen and oxygen. This things had to pump a swimming pool of fuel every few seconds and were beyond anything anyone had done before. Yet they had to fit into a specific space and last 25 flights or more. But what happened was that they pumps could not be built to last more than a flight or two and thus became part of the servicing between every flight. The problem was that they were buried deep inside the engines and were a royal pain to replace. This plus a zillion other similar high level decisions resulted in each shuttle flight turnaround taking forever an costing way too much.
So if you look at the Space X people they are doing the opposite and seeing how good an engine they can build and then plopping a spaceship on top of that. This is how functional companies that don't have too much MBA management bloat engineer things. But my guess is that instead of Boeing just designing a better airplane with composites and seeing what interesting things could be done they made a long series of "executive" decisions and then told outsourced engineering teams to make square pegs fit into round holes. This would be as opposed to a healthy back and fourth where a high level goal is set, the rubber meets the road engineers give their feed back that changes the high level design which results in more feedback until you have a solid high level design that the engineers are fairly certain they can design.
I suspect nearly every programmer here has had a taste of this when some MBA type demands a costly feature that when all is said and done will be used by one person to very little benefit; all because there was no real feedback mechanism to say "whoa there dumb feature."
This. If you get the the bottom of TFA you see what really was driving the decisions about how to design and produce the 787. At the time of the critical decisions for the 787 the head honchos at Boeing were not really Boeing people (a corporation where the key competency for the last 60 years has been the production of profitable commercial airliners.) They came from McDonnell-Douglas, whose key competency was more in the production of military aircraft. The development process of current military hardware is intolerably broken. The old method of subcontracting the design of subsystems and then trying to get them to work together, then just getting more money from Uncle Sam when the result didn't work now results in the aforementioned F22 and F35 (the latter of which may never enter volume production, or at least some variants may not) because complexities have expanded, and costs have likewise increased exponentially. As it turns out, you can't do that with civilian airliners. There aren't friendly Senators and Representatives (whom you have paid off with campaign contributions and subcontractors in their district) to give you more money. And friendly Generals and Admirals (whom are expecting 6 and 7 figure jobs when they retire) who will accept your explanations why things aren't working correctly, and why it's going to be another 3 years to get their gizmo, which doesn't work quite as anticipated. You have shareholders who expect profit, airlines who expect a product in line with what they ordered and expect to pay, and regulators who do not take kindly to aircraft whose electronics bays burst into flames at odd times.
Stop.. In manufacturing there's a lot of integration, third party suppliers or outsourcing as it can also be called. All of those have various degrees of risk associated with them. When you're talking about the scale of what Boeing did on the 787, I think it created new management challenges that they weren't fully expecting and the result was cost overruns and schedule delays. They've always integrated and outsourced with partners. For example I know that they don't make their own nuts and bolts, or rivets or engines for their planes. These come, and have for many many decades, from suppliers who were given specifications and who worked with Boeing. Some of the components such as engines were developed in close partnership, meaning teams from engine manufacturer X at Boeing etc. It's been long since proven that doing it all yourself doesn't get you ahead, you do have more control ala Henry Ford and the River Rouge plant where he didn't have to rely on anybody, or that was his thinking anyway but that went out in the 40s when he couldn't keep up with his contracts for the US government. He even made his own steel. And it eventually became very cumbersome for Ford to maintain this. This was an industry lesson learned and it clearly demonstrated that no manufacturer can exist and create everything on their own and within each business there are associated risks and supply chains that have to be monitored, preserved and nurtured to make it all happen. Do suppliers fail? Yes, but that is one of the risks associated with modern manufacturing and it's up to your business management models to help manage that risk. Obviously in that case Boeing gets a "C-" for the 787 and with the current electrical system woes, they get an "I" for incomplete. They do get an "A" for effort in trying to build something new that hasn't been done before. If you read the stories from people and the press about how their experiences are on the 787, then you'll see what I'm talking about. LCD window shades, quiet cabins.. As somebody who flies, weekly, this is long overdue and it's innovative and yes, with all new innovations there are teething problems. As for the supply chain issues, they'll get settled and yes there may be a "labor" component involved here especially since there have been problems for them getting things manufactured here in the US. If you want a root cause for "outsourcing" look to the US government that in "some" cases goes way out of its way to make things hard for businesses and also creates nice big loopholes in legislation that allow H1-B visas to be used for "Kindergarten Teachers."
Harrison's Postulate - "For every action there is an equal and opposite criticism"
I do manufacture things, and it took me 10 years to figure out how to spec things out so that the techs make exactly what I want.
We've told engineers exactly how to specify products such that they get what they want and most of them proceed to ignore us. For most products we make I can send you a well formatted spreadsheet and if you fill it out completely you'll get exactly the product you want from us. It really isn't all that complicated but does require a certain attention to detail which seems to be lacking.
The technical problem is solved. The human problem maybe isn't.
There is no maybe about it. The human component is not solved the problems are not separable. Humans design things and humans are the ones that make the bad drawings.
I'd have thought aerospace companies are better than someone who has no clue and a decade to learn it on his own, with nobody else to talk to.
We've built parts for aerospace companies. In my experience they are at most marginally better than engineers in other industries. I've actually worked at Boeing while in school and while they have some very smart people working their corporate work culture is not heavy on collaboration. (That's a nice way of saying they don't cooperate well with others)
Boeing managers should have made sure that those engineers from various suppliers do in fact talk to each other, and talk often.
Right in the era of NDAs and intellectual property being regarded as our chief output, that is probably impossible.
Repeal the 17th Amendment TODAY! Also Please Read http://www.gnu.org/philosophy/right-to-read.html
OMG what a bunch of BS. If you have a cellphone anywhere from 75% to 100% is manufacturing "outsourced" and in the case of design that can be anywhere from 0% to 100% as the design is outsourced as well. Look at all the products out there that are the same but sold under different brand names. We're not talking creamed corn here either. Unfortunately the modern world requires outsourcing, not to be confused with shipping jobs overseas because of cheap labor, that's another argument. It's well documented that Boeing met with resistance when it wanted to build a plant in a right to work state (remember strikes at Boeing shutting things down awhile back?) So they invest $750 million in a plant in South Carolina to transition work from Seattle to the new facility. The National Labor Relations Board, a group of political appointees brought suit against them doing it. Now, if Boeing completely "offshored" this plant in Taiwan let's say, what could the NLRB do? Not much but Boeing does a lot of business with the US government (defense contracting) but I bring this up as a prime case where government interference, read blatant union pandering, causes jobs to get shipped overseas. Unions have a place, fair and equitable job markets have a place and that's where the policy makers should be focused. Shit, recently American Airlines signed an agreement with Republic Airlines to fly some of their routes that are serviced by American Eagle. You see because of their labor agreements with pilots unions they were prohibited in flying aircraft with more than 50 seats on Eagle. Think about it, a regional carrier can't buy an Embraier 175 for example because it has too many seats because of a labor agreement and an example as to why organized labor in this country is failing, miserably. I've worked in Union shops, I've been a member of a Union and frankly, the Union's I worked for at the time were more interested in their dues and their bureaucracy than the workers they represented. You may have the right to bargain collectively, organize a union but that shouldn't handcuff how exactly a business delivers services to remain competitive. Otherwise these nice businesses go into Bankruptcy Court where a nice judge who's sympathetic about keeping the business going says "Sure, invalidated those nasty contracts." Then in the case of Hostess, American Airlines and any other company who has leveraged this the workers, the employees get screwed. There needs to be protection for workers rights, there needs to be collective bargaining but a business exists to make a profit and when the labor force becomes problematic in that enterprise, businesses either find a way to work it out, they offshore, they outsource or they go bankrupt.
Harrison's Postulate - "For every action there is an equal and opposite criticism"
Where your analogy breaks down is that weight matters for a plane. Plutonium doesn't make for a superior mousepad, but lithium ion has the advantage of a wider operating temperature range(specifically colder), high efficiency, power capacity, and energy density.
IE a lithium ion battery is going to generally be at least half the weight of any other chemistry for the necessary power/energy demands of the application, and you don't need to worry as much about heating them. For a plane, this makes you really want to use them.
Sure, you might only be saving 100 pounds for the battery pack, less than the added weight an extra obese passenger, but a couple hundred pounds for the batteries, a few hundred from the landing gear, engines, galley, cockpit, seats, wings, flaps, electrical systems, pumps, etc... Next thing you know, your plane is 10-20% lighter and that translates into extra speed, lower fuel cost, etc...
I don't read AC A human right