Excessive Modularity Hindered Development of the 787
TAGmclaren writes "The Harvard Business Review is running a fascinating article exploring the issues facing Boeing's Dreamliner. Rather than simply blaming outsourcing, as much of the commentary has been focused on, the article delves into the benefits of integration and how being integrated when developing a new product gives engineers more degrees of freedom. From the article: 'Historically, Boeing understood that, and had worked with its subcontractors on that basis. If it was going to rely on them, it would provide them with detailed blueprints of the parts that were required — after Boeing had already created them. That, in turn, meant that Boeing had to design all the relevant pieces of the puzzle itself, first. But with the 787, it appears that Boeing tried a very different approach: rather than having the puzzle solved and asking the suppliers to provide a defined puzzle piece, they asked suppliers to create their own blueprints for parts. The puzzle hadn't been properly solved when Boeing asked suppliers for the pieces. It should come as little surprise then, that as the components came back from far-flung suppliers, for the first plane ever made of composite materials... those parts didn't all fit together. Time and cost blew out accordingly. It's easy to blame the outsourcing. But, in this instance, it wasn't so much the outsourcing, as it was the decision to modularize a complicated problem too soon.'"
So Boeing told the contractors what they needed to build, but didn't give them hard specifications? What the hell? Two things:
Boeing needs to have their collective asses kicked for doing it this way, and:
The subcontractors should never have agreed to the work without specs first.
The first one is probably the result of Boeing not wanting to spend the engineering dollars to develop the blueprints, and the second is due to the enormous amounts of money involved in making the parts.
Now that I know this, you'll never catch me on one of those abominations. What the hell was Boeing thinking?
Never underestimate the power of stupid people in large groups.
Yeah baby!!
I suspect that with the mindset of a government contractor they told a bunch of different places sort of what they wanted, but not where it had to be or what it was going to plug into. Measurements, plugs, protocol standards, bolt holes, shape - they all matter - and "on a plane" doesn't answer most of the questions.
The preceding post was not a Slashvertisement.
Systems design in engineering basically involves drawing a box around a bunch of parts and saying "this is a system". The interfaces after that are hopefulyl clean -- good systems design does that, but implicit in the choice of a system breakdown is efficiency loss. I might not, for example, think about the fact that the giant engine at the heart of my car could also run heating. There's this long term conflict in engineering between the need to abstract, which enables all forms of delegation, including outsourcing, subcontracting and even building teams, and the loss of efficiency. Good engineers learn things at an almost inexpressible level,developing jargons for the systems under their purview -- in the case of Boeing, there was literally one guy who was their expert on cabling. If you wanted to submit a drawing change, he could envision the change in the cabling of the plane and whether the change was physically possible. That's always been the bane of system abstraction - you find these things that have to cross systems and, if you don't recognize them early enough, they come back to bite you in all sorts of creative ways. Kelly Johnson was a big believer in this. His rules for skunkworks explicitly required that engineers had to be within a specific number of feet of the shop floor -- that way they weren't too divorced from the reality of the products they were making. You see this in the design of a lot of the early computer systems as well, parts bolted together in weird ways before we started developing this high-level view of what systems actually made up a computer.
This is what happens with subcontracting you give up to much control and people in the contracting line cut corners where they can.
Obviously, Boeing should simply have specified that all the contractors deliver components that accept and output plaintext, and then used pipes and awk to cobble the pieces together into a working system! What could possibly go wrong?
... when you are a big top secret defense contractor and you attempt to unify your development processes across all of your business units to "save money" through homogenization.
You can design something as an "interconnected series of black boxes" when it's something simple like a missile.
A 787 and other development abortions like the F22 and F35 are infinitely more complex than a simple war munition, and cannot be properly designed as an "interconnected series of black boxes."
By "hindered development," do they mean "made it harder?" If so...fucking duh. That's what happens when you try a new approach to building something...but that's not necessarily a reason not to innovate, and the fact that mistakes were made isn't necessarily an indictment of the activities that took place in the course of that innovation.
Coming up with a new way of building a large commercial airliner is not going to be easy, and you're going to make mistakes. The article seems a little light on details; I don't buy the notion that Boeing simply told their subcontractors, without any details whatsoever, to build components. I would wager that the real truth is that the subcontractors were given specifications with regard to specific points of integration, but that Boeing underestimated the potential to stay within those specs and still deliver a component that was incompatible with the surrounding area of the aircraft. Whoops, mistake...but you make mistakes when you innovate, and then you learn and move forward. This article seems to imply that Boeing has no idea how to build aircraft at all, and that's just not true.
For your security, this post has been encrypted with ROT-13, twice.
Boeing didn't want to hire all the engineers needed to design the 787. So when they outsourced these subsystems they also counted on their suppliers to do the engineering of these subsystems.
The problem is that engineers are not fungible. Boeing didn't appreciate this, any more than the software industry did when it started outsourcing.
An aerospace structural frame engineer is not the same thing as a marine structure engineer. There are huge differences in the body of experience despite the fact that they both use the same tools.
This was the primary cause of the delays Boeing had. It will continue to be a problem for anyone who tries this sort of outsourcing.
Wow, they crowd-sourced the design of a wide body aircraft?
So it's their reputation on the line, and likely a lot of the legal liability .. but they gave the suppliers reign to design their own parts?
That sounds like an epic fail in engineering to me. The 777 was a marvel in that every part had been designed and modeled in a computer before they ever built anything -- this sounds like a hodge-podge of parts.
At around $200 million a pop or so, that sounds awfully risky.
Lost at C:>. Found at C.
It wasn't outsourcing the production, it was outsourcing the design that was the problem. Particularly outsourcing the design of subsystems to different suppliers. You don't know if the pieces will work together until they come back home. It's not like an engineer at supplier 1 can walk down the hall and talk to a guy working on a part at supplier 2, but this does happen when you're all under one roof.
Was not a Beech ðe first composite airplane?
Leandro Guimarães Faria Corcete DUTRA
DA, DBA, SysAdmin, Data Modeller
GNU Project, Debian GNU/Lin
From the BBC story:
"I think people had their fingers crossed that it was a battery fault... it looks more systemic and serious to me. I suspect it could be difficult to identify the cause," [Keith Hayward, head of research at the Royal Aeronautical Society] said.
I would hope the folks in change of designing and building aircraft would depend on measurements and calculations, not crossed fingers. Did they also consult a Ouija board?
Burt Rutan's beautiful creation holds that title.
http://en.wikipedia.org/wiki/Beechcraft_Starship
- Preferences: Solaris 10 (servers), Ubuntu (desktops), Solaris 11 (personal servers) -
In it, he termed the architecture of the GS Yuasa battery packs supplied to Boeing "inherent unsafe," and predicted more fires from the same causes due to its design.
Specifically, Musk criticized the use of large-format lithium-ion cells "without enough space between them to isolate against the cell-to-cell thermal domino effect."
He also noted that when thermal runaway occurs in the larger cells, more energy is released by the single cell than comes from a small-format "commodity" cell, of the type used by the thousands in Tesla battery packs.
And he went on to highlight what he viewed as the dangers of batteries using those large-format cells, saying they have a "fundamental safety issue" because it's harder to keep the internal temperature of a large-format cell consistent from the center to the edges.
Not surprisingly, Mike Sinnett--Boeing's chief engineer for the 787 project--counters that the company designed the pack to cope with not only a single cell failure but to contain runaway thermal events as well."
http://www.greencarreports.com/news/1082007_tesla-ceo-musk-boeing-787-batteries-inherently-unsafe
It's easy to blame the outsourcing.
Ok so if it wasn't outsourcing what was the problem?
But, in this instance, it wasn't so much the outsourcing, as it was the decision to modularize a complicated problem too soon.'
Oh, so it was outsourcing, they were just trying to outsource as early as possible so they won't have to pay engineers to develop the specs.
There was a short article on the Dreamliner in the latest New Yorker magazine REQUIEM FOR A DREAMLINER? . Quote Surowiecki :The Dreamliner was supposed to become famous for its revolutionary design. Instead, it’s become an object lesson in how not to build an airplane.
To understand why, you need to go back to 1997, when Boeing merged with McDonnell Douglas. Technically, Boeing bought McDonnell Douglas. But, as Richard Aboulafia, a noted industry analyst with the Teal Group, told me, “McDonnell Douglas in effect acquired Boeing with Boeing’s money.” McDonnell Douglas executives became key players in the new company, and the McDonnell Douglas culture, averse to risk and obsessed with cost-cutting, weakened Boeing’s historical commitment to making big investments in new products. Aboulafia says, “After the merger, there was a real battle over the future of the company, between the engineers and the finance and sales guys.” The nerds may have been running the show in Silicon Valley, but at Boeing they were increasingly marginalized by the bean counters.
Read more: http://www.newyorker.com/talk/financial/2013/02/04/130204ta_talk_surowiecki#ixzz2JTGx7SPc
Post-web people may like to read what was all the rage in the early 1990's: the deep philosophy of software development. Doug Lea has a concise summary, so you don't have to read several hundred pages. And here's a quick direct quote from Christopher Alexander I just now Googled:
Elon Musk of Tesla Motors who has offered to sell batteries to Boeing agrees with the un-safeness of these competitors batteries.
There you go, a bit of clarification for you.
Boeing built it in pieces because they had to be able to sell the plane to foreign countries. It's a tremendous sales aid being able to point to parts of the plane being built by the country interested in purchasing the plane. This is why it's a mess of a build.
TLDR scalability issue due to cubed vs square law scaling.
High temperature can blow up some Li cells. The total thermal energy scales with volume, but the surface heat can escape from scales with surface area. So its hard (although not impossible) for one cell in a laptop to blow up the adjacent cells. But if you make the individual cells big enough you can get a chain reaction going.
This is a meta issue anyway. There are battery techs not susceptible to chain reactions, and not susceptible to occasionally blowing up an individual cell anyway.
Lets say you decide to make mousepads out of fissionable plutonium. Boss reports that they're getting bad PR because of mfgr variations or mishandling sometimes the plutonium mousepad blows up, although you tried to design it for a very unfavorable geometry. Well you can argue geometry and mfgr tolerances all day, but the mistake was making the mousepad out of plutonium, not making it "wrong".
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
Well, demonstrably it didn't contain much because there was smoke on board. Containment means the battery dies, and outside, apart from lost functionality, nothing bad happens. Smoke is kind of a no-no.
A successful API design takes a mixture of software design and pedagogy.
I'm a systems engineer,
Off-topic - what's your education and training?
In this artcle they cite a 2001 Boeing paper which I find very interesting.
All those MBA bosses should have a look, it seems very few have learned any lesson since then.
The point is made that not only is the work out-sourced; all the profits associated with the work are out-sourced, too.
...
A strong warning is included about the perils of sub-optimum solutions in which individual cost are minimized in isolation.
It is quite enlightening, given that their problem today could very likely come from those interdepartmental interactions not thoroughly planned enough.
I have news for you. This is how most things are developed.
Homes are built from standardized components, as are production machines, cars, computers and even software. Often, we pick our components first, with only a hazy idea of the finished product. We think we know what we are developing, but after its built, it usually looks quite different than first imagined.
The key is Boeing is developing a much more difficult design and this is their first attempt at using this method. It is understandable that things didn't work perfectly, just like the first time we used CAD systems - something we have come to depend on.
Aircraft today is far more complex then before, and its getting more complex. I can see why Boeing is attempting to spread the workload across a virtual company, rather than attempt to do it completely in-house.
Place nail here >+
When it comes to mechanical parts, geometric dimensioning and tolerancing is a solved problem. When it comes to electrical interoperability, one'd think that's a solved problem as well.
"Solved problem"? HAAHAHAHHHAAHHAAAAA....
You don't manufacture things for a living do you? I run a company that makes wire harnesses. We're a contract manufacturer - we don't design things, we just take prints and build what is on the prints. I can count on my fingers on one hand the number of prints we have gotten from customers which were correct and sufficiently detailed such that the product could be built without asking any questions. There pretty much always are critical details left out of the prints. About 2/3 of the prints we see have incompatible parts specified. About half are missing at least one important dimension such as length. About 10% have missing parts and about 25% have incompatible parts. About 20% specify needlessly expensive parts like gold plated terminals that cost more but provide no actual performance benefit. Most of them leave off at least one critical tolerance. I've even seen drawings with dimension in inches and tolerances in metric.
Why does this happen? For the most part because an alarming number of engineers doing the drawings aren't actually very good at their job. Some of them are just plain lazy. The electrical engineers usually can specify a wire schematic but often have no idea whether something can actually be built or know much about industry standards. The more mechanical engineers (yes mechanical engineers can and do design circuits) tend to create bad designs and specify the wrong parts because they don't know any better. Sometimes they are trying to do a good job but they don't bother to consult manufacturing during the design process and they come up with a stupid design or something that is impossible to build.
I have run into some good engineers but they are the exception.
Obviously a large project has to have an overarching design and direction but a great example of a failed top down aviation design would be the Space Shuttle. They designed many of the larger systems in oddly specific waves of a wand and then left it to engineers to actually invent them. A really great example of this failure were the cryopumps for the liquid hydrogen and oxygen. This things had to pump a swimming pool of fuel every few seconds and were beyond anything anyone had done before. Yet they had to fit into a specific space and last 25 flights or more. But what happened was that they pumps could not be built to last more than a flight or two and thus became part of the servicing between every flight. The problem was that they were buried deep inside the engines and were a royal pain to replace. This plus a zillion other similar high level decisions resulted in each shuttle flight turnaround taking forever an costing way too much.
So if you look at the Space X people they are doing the opposite and seeing how good an engine they can build and then plopping a spaceship on top of that. This is how functional companies that don't have too much MBA management bloat engineer things. But my guess is that instead of Boeing just designing a better airplane with composites and seeing what interesting things could be done they made a long series of "executive" decisions and then told outsourced engineering teams to make square pegs fit into round holes. This would be as opposed to a healthy back and fourth where a high level goal is set, the rubber meets the road engineers give their feed back that changes the high level design which results in more feedback until you have a solid high level design that the engineers are fairly certain they can design.
I suspect nearly every programmer here has had a taste of this when some MBA type demands a costly feature that when all is said and done will be used by one person to very little benefit; all because there was no real feedback mechanism to say "whoa there dumb feature."
is the problem. Well, you save a few pennies in development Boeing, hows that working out for you?
The Kruger Dunning explains most post on
They should have hired Linus Torvalds to consult and used git for the master blueprint.
*** Don't be dull.***
The bottom line of this story is a mistake that is a peril to virtually all large complex systems -- in particular (of interest to this forum), software-based systems. I see this mistake made in large complex (are there any other kind?) military systems, especially distributed ones -- my primary field of professional expertise. The context for this mistake often is that the system integrator usually prematurely farms out system components to subcontractors in as many politically important states and subcontractors as possible. Moreover, there is a DoD bandwagon called Modular Open Systems Architecture, which (like capitalism, as we have seen all too clearly) can be misapplied and exaggerated. Wouldn't I just love to describe some (very expensive) real examples .... but I need my consulting clients and security clearance :)
Doug Jensen
I do manufacture things, and it took me 10 years to figure out how to spec things out so that the techs make exactly what I want.
We've told engineers exactly how to specify products such that they get what they want and most of them proceed to ignore us. For most products we make I can send you a well formatted spreadsheet and if you fill it out completely you'll get exactly the product you want from us. It really isn't all that complicated but does require a certain attention to detail which seems to be lacking.
The technical problem is solved. The human problem maybe isn't.
There is no maybe about it. The human component is not solved the problems are not separable. Humans design things and humans are the ones that make the bad drawings.
I'd have thought aerospace companies are better than someone who has no clue and a decade to learn it on his own, with nobody else to talk to.
We've built parts for aerospace companies. In my experience they are at most marginally better than engineers in other industries. I've actually worked at Boeing while in school and while they have some very smart people working their corporate work culture is not heavy on collaboration. (That's a nice way of saying they don't cooperate well with others)
Maybe they should have had more daily scrums in their agile development process...
By the taping of my glasses, something geeky this way passes
Have gnu, will travel.
Pack the batteries in sugar, that way when they burn you get pretty colors and nice caramel smell.
I only look human.
My mother is a halfling and my dad is an ogre, so that makes me an Ogreling
OMG what a bunch of BS. If you have a cellphone anywhere from 75% to 100% is manufacturing "outsourced" and in the case of design that can be anywhere from 0% to 100% as the design is outsourced as well. Look at all the products out there that are the same but sold under different brand names. We're not talking creamed corn here either. Unfortunately the modern world requires outsourcing, not to be confused with shipping jobs overseas because of cheap labor, that's another argument. It's well documented that Boeing met with resistance when it wanted to build a plant in a right to work state (remember strikes at Boeing shutting things down awhile back?) So they invest $750 million in a plant in South Carolina to transition work from Seattle to the new facility. The National Labor Relations Board, a group of political appointees brought suit against them doing it. Now, if Boeing completely "offshored" this plant in Taiwan let's say, what could the NLRB do? Not much but Boeing does a lot of business with the US government (defense contracting) but I bring this up as a prime case where government interference, read blatant union pandering, causes jobs to get shipped overseas. Unions have a place, fair and equitable job markets have a place and that's where the policy makers should be focused. Shit, recently American Airlines signed an agreement with Republic Airlines to fly some of their routes that are serviced by American Eagle. You see because of their labor agreements with pilots unions they were prohibited in flying aircraft with more than 50 seats on Eagle. Think about it, a regional carrier can't buy an Embraier 175 for example because it has too many seats because of a labor agreement and an example as to why organized labor in this country is failing, miserably. I've worked in Union shops, I've been a member of a Union and frankly, the Union's I worked for at the time were more interested in their dues and their bureaucracy than the workers they represented. You may have the right to bargain collectively, organize a union but that shouldn't handcuff how exactly a business delivers services to remain competitive. Otherwise these nice businesses go into Bankruptcy Court where a nice judge who's sympathetic about keeping the business going says "Sure, invalidated those nasty contracts." Then in the case of Hostess, American Airlines and any other company who has leveraged this the workers, the employees get screwed. There needs to be protection for workers rights, there needs to be collective bargaining but a business exists to make a profit and when the labor force becomes problematic in that enterprise, businesses either find a way to work it out, they offshore, they outsource or they go bankrupt.
Harrison's Postulate - "For every action there is an equal and opposite criticism"
Didn't some Fiskers blow up with the flooding in Hurricane Sandy? So now you want to put these on an airplane as a critical part of the electrical system. Flawed thinking. Go back to a stable design and look for the 200lbs of weight savings elsewhere. Hire thinner flight attendants or don't fly fat people around.
Harrison's Postulate - "For every action there is an equal and opposite criticism"
The Space Shuttle Main Engine was handled in a different manner, top down, we might say. The engine was designed and put together all at once with relatively little detailed preliminary study of the material and components. Then when troubles are found in the bearings, turbine blades, coolant pipes, etc., it is more expensive and difficult to discover the causes and make changes. For example, cracks have been found in the turbine blades of the high pressure oxygen turbopump. Are they caused by flaws in the material, the effect of the oxygen atmosphere on the properties of the material, the thermal stresses of startup or shutdown, the vibration and stresses of steady running, or mainly at some resonance at certain speeds, etc.? How long can we run from crack initiation to crack failure, and how does this depend on power level? Using the completed engine as a test bed to resolve such questions is extremely expensive. One does not wish to lose an entire engine in order to find out where and how failure occurs. Yet, an accurate knowledge of this information is essential to acquire a confidence in the engine reliability in use. Without detailed understanding, confidence can not be attained. A further disadvantage of the top-down method is that, if an understanding of a fault is obtained, a simple fix, such as a new shape for the turbine housing, may be impossible to implement without a redesign of the entire engine.
Full report is here http://www.ralentz.com/old/space/feynman-report.html
I think part of the problem is that Boeing defined what components they needed, but not necessarily completely how they were to be built or function internally (for things that have internals). For example, they may have specified an electronic component by its inputs/outputs and working/environmental tolerances, but not anything about the internals. In theory, this "black box" approach should work pretty well, but - as we programmers know - side effects, edge conditions and unknowns are a bitch.
In the case of the Li-Ion battery and its external monitoring/charging equipment, different assumptions may have been made on either side. As for other components, sometimes one needs to know how they're to be assembled to make them correctly.
It must have been something you assimilated. . . .
Where your analogy breaks down is that weight matters for a plane. Plutonium doesn't make for a superior mousepad, but lithium ion has the advantage of a wider operating temperature range(specifically colder), high efficiency, power capacity, and energy density.
IE a lithium ion battery is going to generally be at least half the weight of any other chemistry for the necessary power/energy demands of the application, and you don't need to worry as much about heating them. For a plane, this makes you really want to use them.
Sure, you might only be saving 100 pounds for the battery pack, less than the added weight an extra obese passenger, but a couple hundred pounds for the batteries, a few hundred from the landing gear, engines, galley, cockpit, seats, wings, flaps, electrical systems, pumps, etc... Next thing you know, your plane is 10-20% lighter and that translates into extra speed, lower fuel cost, etc...
I don't read AC A human right
I know I'm late for a nice flamefest, but I'm sure someone will point out how Krugman was wrong about this when he discussed it almost two years ago: http://krugman.blogs.nytimes.com/2011/02/19/thank-you-boeing/
No, the problem is bureaucracy. Bureaucratic systems inefficient at management, partially due to a misunderstanding of and over-reliance upon digital data.
in other words: BAD MANAGEMENT
You attempted to give examples of "(IIRC) three" unspecified aerospace projects that had been prototyped using all-digital, proving nothing. Assuming your examples are real, absence of evidence is not evidence of absence. Aerospace prototyping failures that result in commercial product failures are a big fucking deal.
Look, here's what you are missing: When human lives are involved, prototyping things like...um...the F-22 or this 787 with a real-life prototype is *absolutely necessary*
The only way to conclude otherwise is to de-value human life. Do all the actuarial MBA-bullshit spreadsheets you want, you are still doing less work, less science on a complex project purely to save money *at the expense of lives*
Research 'military boondoggles' and you'll see Boeing has experience with this. The F-22 Raptor, be it digital-only prototype or not, is an example of the same **FAILURE OF MANAGEMENT** in all ways.
I'm not just talking 'operational' failure. There is a systemic problem and a systemic inability to do proper error correction...so it's like having clogged pipes that are also rusting.
That's the problem here. A fundamental flaw that with certainty will result in using bad metrics in the lab when deciding the statistical tolerances for the prototyping program....that's just one mistake.
Thank you Dave Raggett
Did they NOT have fires during the thousands of hours of test-flights?
These are my friends, See how they glisten. See this one shine, how he smiles in the light.
I'm calling troll....
I refuted your points, with specific examples using technical language (not 'buzzwords'). I gave the F-22 example. I used systems science and 'management science' terminology.
I offered a simple thesis...your approach is a cost-saving measure but ill-suited to situation like vehicles where human lives are at stake. I said your method sacrificed human lives for 'efficiency'.
That is a falsify-able statement you could have argued against...if you weren't a TROLL
Anyone reading this deep can judge but I deem thee a troll and you have rendered this part of the thread irrelevant.
Thank you Dave Raggett
Chair, President, CEO - BA, MBA (not a shred of actual engineering experience) President, CEO - BA, MBA (used to be a 727 mechanic . . . in the 70s) Senior VP - history major Senior VP - CIVIL engineering and a master's in engineering management . . . Dilbert says hello) CTO - BS/MA physics, PhD engineering I'm not arguing only aeronautical engineers could run an airplane manufacturer. That would be similar to saying only doctors should run hospitals/healthcare companies. BUT . . . if you accumulate too many 'profit' bean counters, an organization can often lose sight of the fact that it's the product that makes money not making money off the product. The only reason we get away with it in healthcare is because society pays us to do stuff . . . regardless of whether we do it well. "What's the difference between a doctor and a pilot?" A doctor isn't going to go down with you. -David Healy, MD Arguably the same could be said for senior management at most US corporations.