Slashdot Mirror
Excessive Modularity Hindered Development of the 787
TAGmclaren writes "The Harvard Business Review is running a fascinating article exploring the issues facing Boeing's Dreamliner. Rather than simply blaming outsourcing, as much of the commentary has been focused on, the article delves into the benefits of integration and how being integrated when developing a new product gives engineers more degrees of freedom. From the article: 'Historically, Boeing understood that, and had worked with its subcontractors on that basis. If it was going to rely on them, it would provide them with detailed blueprints of the parts that were required — after Boeing had already created them. That, in turn, meant that Boeing had to design all the relevant pieces of the puzzle itself, first. But with the 787, it appears that Boeing tried a very different approach: rather than having the puzzle solved and asking the suppliers to provide a defined puzzle piece, they asked suppliers to create their own blueprints for parts. The puzzle hadn't been properly solved when Boeing asked suppliers for the pieces. It should come as little surprise then, that as the components came back from far-flung suppliers, for the first plane ever made of composite materials... those parts didn't all fit together. Time and cost blew out accordingly. It's easy to blame the outsourcing. But, in this instance, it wasn't so much the outsourcing, as it was the decision to modularize a complicated problem too soon.'"
So Boeing told the contractors what they needed to build, but didn't give them hard specifications? What the hell? Two things:
Boeing needs to have their collective asses kicked for doing it this way, and:
The subcontractors should never have agreed to the work without specs first.
The first one is probably the result of Boeing not wanting to spend the engineering dollars to develop the blueprints, and the second is due to the enormous amounts of money involved in making the parts.
Now that I know this, you'll never catch me on one of those abominations. What the hell was Boeing thinking?
Never underestimate the power of stupid people in large groups.
Yeah baby!!
I'm pretty sure the 787 wasn't "the first plane ever made of composite materials". The first BIG plane maybe, but not the first plane.
I suspect that with the mindset of a government contractor they told a bunch of different places sort of what they wanted, but not where it had to be or what it was going to plug into. Measurements, plugs, protocol standards, bolt holes, shape - they all matter - and "on a plane" doesn't answer most of the questions.
The preceding post was not a Slashvertisement.
Systems design in engineering basically involves drawing a box around a bunch of parts and saying "this is a system". The interfaces after that are hopefulyl clean -- good systems design does that, but implicit in the choice of a system breakdown is efficiency loss. I might not, for example, think about the fact that the giant engine at the heart of my car could also run heating. There's this long term conflict in engineering between the need to abstract, which enables all forms of delegation, including outsourcing, subcontracting and even building teams, and the loss of efficiency. Good engineers learn things at an almost inexpressible level,developing jargons for the systems under their purview -- in the case of Boeing, there was literally one guy who was their expert on cabling. If you wanted to submit a drawing change, he could envision the change in the cabling of the plane and whether the change was physically possible. That's always been the bane of system abstraction - you find these things that have to cross systems and, if you don't recognize them early enough, they come back to bite you in all sorts of creative ways. Kelly Johnson was a big believer in this. His rules for skunkworks explicitly required that engineers had to be within a specific number of feet of the shop floor -- that way they weren't too divorced from the reality of the products they were making. You see this in the design of a lot of the early computer systems as well, parts bolted together in weird ways before we started developing this high-level view of what systems actually made up a computer.
This is what happens with subcontracting you give up to much control and people in the contracting line cut corners where they can.
The reality is -- nobody knows how Lithium batteries actually work. We just know that they work. And they spontaneously combust. Stay away from electric cars and airplanes if they use lithium batteries.
Obviously, Boeing should simply have specified that all the contractors deliver components that accept and output plaintext, and then used pipes and awk to cobble the pieces together into a working system! What could possibly go wrong?
... when you are a big top secret defense contractor and you attempt to unify your development processes across all of your business units to "save money" through homogenization.
You can design something as an "interconnected series of black boxes" when it's something simple like a missile.
A 787 and other development abortions like the F22 and F35 are infinitely more complex than a simple war munition, and cannot be properly designed as an "interconnected series of black boxes."
By "hindered development," do they mean "made it harder?" If so...fucking duh. That's what happens when you try a new approach to building something...but that's not necessarily a reason not to innovate, and the fact that mistakes were made isn't necessarily an indictment of the activities that took place in the course of that innovation.
Coming up with a new way of building a large commercial airliner is not going to be easy, and you're going to make mistakes. The article seems a little light on details; I don't buy the notion that Boeing simply told their subcontractors, without any details whatsoever, to build components. I would wager that the real truth is that the subcontractors were given specifications with regard to specific points of integration, but that Boeing underestimated the potential to stay within those specs and still deliver a component that was incompatible with the surrounding area of the aircraft. Whoops, mistake...but you make mistakes when you innovate, and then you learn and move forward. This article seems to imply that Boeing has no idea how to build aircraft at all, and that's just not true.
For your security, this post has been encrypted with ROT-13, twice.
It is very easy to criticize the 787 and its development plan. It was a very bold business and engineering project and necessitated some very risky decision making. In many ways Boeing threw out everything it knew about manufacturing planes to use new materials that would give it a dramatic advantage over its competition. Management basically staked their company's long-term viability to this one massive project spanning more than a decade.
The project was visionary and most companies wouldn't have had the balls to pull the trigger. In my mind, the question wasn't whether some things would go wrong but how many. You don't push the bleeding edge without making some errors. If those errors can be contained and damages mitigated, wonderful.
So now we get some folks with 20:20 hindsight saying that management approach x caused problem y. My response: no crap. Every approach results in some problem. But if Boeing tried an integrated approach there would have been a whole slew of other problems including a lack of accountability and transparency.
Boeing has figured out a lot of engineering problems through this project and has changed the way planes of the future will be built. They should be commended for this. There are new industries that have been and are being built around these new manufacturing process. Boeing will take the pains of being on the front of this, but will also get the rewards as well. Let the 'analysts' have their fun, but they are not the ones taking the risk and changing the world.
-- MyLongNickName
Boeing didn't want to hire all the engineers needed to design the 787. So when they outsourced these subsystems they also counted on their suppliers to do the engineering of these subsystems.
The problem is that engineers are not fungible. Boeing didn't appreciate this, any more than the software industry did when it started outsourcing.
An aerospace structural frame engineer is not the same thing as a marine structure engineer. There are huge differences in the body of experience despite the fact that they both use the same tools.
This was the primary cause of the delays Boeing had. It will continue to be a problem for anyone who tries this sort of outsourcing.
This is what happens when you have too many MBAs in charge who think that enginneering experience is also modular and can be replaced. Ironically it is really the MBAs who are the most modular and expendable.
Wow, they crowd-sourced the design of a wide body aircraft?
So it's their reputation on the line, and likely a lot of the legal liability .. but they gave the suppliers reign to design their own parts?
That sounds like an epic fail in engineering to me. The 777 was a marvel in that every part had been designed and modeled in a computer before they ever built anything -- this sounds like a hodge-podge of parts.
At around $200 million a pop or so, that sounds awfully risky.
Lost at C:>. Found at C.
It wasn't outsourcing the production, it was outsourcing the design that was the problem. Particularly outsourcing the design of subsystems to different suppliers. You don't know if the pieces will work together until they come back home. It's not like an engineer at supplier 1 can walk down the hall and talk to a guy working on a part at supplier 2, but this does happen when you're all under one roof.
Was not a Beech ðe first composite airplane?
Leandro Guimarães Faria Corcete DUTRA
DA, DBA, SysAdmin, Data Modeller
GNU Project, Debian GNU/Lin
From the BBC story:
"I think people had their fingers crossed that it was a battery fault... it looks more systemic and serious to me. I suspect it could be difficult to identify the cause," [Keith Hayward, head of research at the Royal Aeronautical Society] said.
I would hope the folks in change of designing and building aircraft would depend on measurements and calculations, not crossed fingers. Did they also consult a Ouija board?
Burt Rutan's beautiful creation holds that title.
http://en.wikipedia.org/wiki/Beechcraft_Starship
- Preferences: Solaris 10 (servers), Ubuntu (desktops), Solaris 11 (personal servers) -
It's easy to blame the outsourcing.
Ok so if it wasn't outsourcing what was the problem?
But, in this instance, it wasn't so much the outsourcing, as it was the decision to modularize a complicated problem too soon.'
Oh, so it was outsourcing, they were just trying to outsource as early as possible so they won't have to pay engineers to develop the specs.
There was a short article on the Dreamliner in the latest New Yorker magazine REQUIEM FOR A DREAMLINER? . Quote Surowiecki :The Dreamliner was supposed to become famous for its revolutionary design. Instead, it’s become an object lesson in how not to build an airplane.
To understand why, you need to go back to 1997, when Boeing merged with McDonnell Douglas. Technically, Boeing bought McDonnell Douglas. But, as Richard Aboulafia, a noted industry analyst with the Teal Group, told me, “McDonnell Douglas in effect acquired Boeing with Boeing’s money.” McDonnell Douglas executives became key players in the new company, and the McDonnell Douglas culture, averse to risk and obsessed with cost-cutting, weakened Boeing’s historical commitment to making big investments in new products. Aboulafia says, “After the merger, there was a real battle over the future of the company, between the engineers and the finance and sales guys.” The nerds may have been running the show in Silicon Valley, but at Boeing they were increasingly marginalized by the bean counters.
Read more: http://www.newyorker.com/talk/financial/2013/02/04/130204ta_talk_surowiecki#ixzz2JTGx7SPc
Post-web people may like to read what was all the rage in the early 1990's: the deep philosophy of software development. Doug Lea has a concise summary, so you don't have to read several hundred pages. And here's a quick direct quote from Christopher Alexander I just now Googled:
Boeing built it in pieces because they had to be able to sell the plane to foreign countries. It's a tremendous sales aid being able to point to parts of the plane being built by the country interested in purchasing the plane. This is why it's a mess of a build.
I'm a systems engineer,
Off-topic - what's your education and training?
... the decision to outsource.
I agree.
In this artcle they cite a 2001 Boeing paper which I find very interesting.
All those MBA bosses should have a look, it seems very few have learned any lesson since then.
The point is made that not only is the work out-sourced; all the profits associated with the work are out-sourced, too.
...
A strong warning is included about the perils of sub-optimum solutions in which individual cost are minimized in isolation.
It is quite enlightening, given that their problem today could very likely come from those interdepartmental interactions not thoroughly planned enough.
I have news for you. This is how most things are developed.
Homes are built from standardized components, as are production machines, cars, computers and even software. Often, we pick our components first, with only a hazy idea of the finished product. We think we know what we are developing, but after its built, it usually looks quite different than first imagined.
The key is Boeing is developing a much more difficult design and this is their first attempt at using this method. It is understandable that things didn't work perfectly, just like the first time we used CAD systems - something we have come to depend on.
Aircraft today is far more complex then before, and its getting more complex. I can see why Boeing is attempting to spread the workload across a virtual company, rather than attempt to do it completely in-house.
Place nail here >+
When it comes to mechanical parts, geometric dimensioning and tolerancing is a solved problem. When it comes to electrical interoperability, one'd think that's a solved problem as well.
"Solved problem"? HAAHAHAHHHAAHHAAAAA....
You don't manufacture things for a living do you? I run a company that makes wire harnesses. We're a contract manufacturer - we don't design things, we just take prints and build what is on the prints. I can count on my fingers on one hand the number of prints we have gotten from customers which were correct and sufficiently detailed such that the product could be built without asking any questions. There pretty much always are critical details left out of the prints. About 2/3 of the prints we see have incompatible parts specified. About half are missing at least one important dimension such as length. About 10% have missing parts and about 25% have incompatible parts. About 20% specify needlessly expensive parts like gold plated terminals that cost more but provide no actual performance benefit. Most of them leave off at least one critical tolerance. I've even seen drawings with dimension in inches and tolerances in metric.
Why does this happen? For the most part because an alarming number of engineers doing the drawings aren't actually very good at their job. Some of them are just plain lazy. The electrical engineers usually can specify a wire schematic but often have no idea whether something can actually be built or know much about industry standards. The more mechanical engineers (yes mechanical engineers can and do design circuits) tend to create bad designs and specify the wrong parts because they don't know any better. Sometimes they are trying to do a good job but they don't bother to consult manufacturing during the design process and they come up with a stupid design or something that is impossible to build.
I have run into some good engineers but they are the exception.
Obviously a large project has to have an overarching design and direction but a great example of a failed top down aviation design would be the Space Shuttle. They designed many of the larger systems in oddly specific waves of a wand and then left it to engineers to actually invent them. A really great example of this failure were the cryopumps for the liquid hydrogen and oxygen. This things had to pump a swimming pool of fuel every few seconds and were beyond anything anyone had done before. Yet they had to fit into a specific space and last 25 flights or more. But what happened was that they pumps could not be built to last more than a flight or two and thus became part of the servicing between every flight. The problem was that they were buried deep inside the engines and were a royal pain to replace. This plus a zillion other similar high level decisions resulted in each shuttle flight turnaround taking forever an costing way too much.
So if you look at the Space X people they are doing the opposite and seeing how good an engine they can build and then plopping a spaceship on top of that. This is how functional companies that don't have too much MBA management bloat engineer things. But my guess is that instead of Boeing just designing a better airplane with composites and seeing what interesting things could be done they made a long series of "executive" decisions and then told outsourced engineering teams to make square pegs fit into round holes. This would be as opposed to a healthy back and fourth where a high level goal is set, the rubber meets the road engineers give their feed back that changes the high level design which results in more feedback until you have a solid high level design that the engineers are fairly certain they can design.
I suspect nearly every programmer here has had a taste of this when some MBA type demands a costly feature that when all is said and done will be used by one person to very little benefit; all because there was no real feedback mechanism to say "whoa there dumb feature."
is the problem. Well, you save a few pennies in development Boeing, hows that working out for you?
The Kruger Dunning explains most post on
They should have hired Linus Torvalds to consult and used git for the master blueprint.
*** Don't be dull.***
The bottom line of this story is a mistake that is a peril to virtually all large complex systems -- in particular (of interest to this forum), software-based systems. I see this mistake made in large complex (are there any other kind?) military systems, especially distributed ones -- my primary field of professional expertise. The context for this mistake often is that the system integrator usually prematurely farms out system components to subcontractors in as many politically important states and subcontractors as possible. Moreover, there is a DoD bandwagon called Modular Open Systems Architecture, which (like capitalism, as we have seen all too clearly) can be misapplied and exaggerated. Wouldn't I just love to describe some (very expensive) real examples .... but I need my consulting clients and security clearance :)
Doug Jensen
Take THAT Andrew Tanenbaum!
I do manufacture things, and it took me 10 years to figure out how to spec things out so that the techs make exactly what I want.
We've told engineers exactly how to specify products such that they get what they want and most of them proceed to ignore us. For most products we make I can send you a well formatted spreadsheet and if you fill it out completely you'll get exactly the product you want from us. It really isn't all that complicated but does require a certain attention to detail which seems to be lacking.
The technical problem is solved. The human problem maybe isn't.
There is no maybe about it. The human component is not solved the problems are not separable. Humans design things and humans are the ones that make the bad drawings.
I'd have thought aerospace companies are better than someone who has no clue and a decade to learn it on his own, with nobody else to talk to.
We've built parts for aerospace companies. In my experience they are at most marginally better than engineers in other industries. I've actually worked at Boeing while in school and while they have some very smart people working their corporate work culture is not heavy on collaboration. (That's a nice way of saying they don't cooperate well with others)
Maybe they should have had more daily scrums in their agile development process...
By the taping of my glasses, something geeky this way passes
Have gnu, will travel.
Pack the batteries in sugar, that way when they burn you get pretty colors and nice caramel smell.
I only look human.
My mother is a halfling and my dad is an ogre, so that makes me an Ogreling
Nvidia designs the chips, and Intel (or IBM) build the chips for them. Nvidia then builds a few dozen reference boards, and ships those to external vendors for manufacture. Nvidia then provides software/firmware. We are talking about aircraft, not graphics processors, but the idea is the same. You engineer at home, and outsource. If you give the subcontractors the job of engineering too, then you must make sure that your specifications allow for external engineering such as "Leave a 6 cubic inch wiring hole in this space frame within 5 inches of the top." Because parts have to mate with other parts, I don't see how that would be easy to do, and doing a computational variance simulation on this would be almost impossible, and without a computational variance simulation, build quality suddenly comes into question. See what happens when you put an MBA with the idea of penny pinching in the place where an engineer should be!
It's fun to watch advocates of outsourcing scramble to prop up their failed ideologies in the face of contradictory evidence. The fact is that Boeing not only outsourced manufacturing, but design and it was a colossal failure.
Can it really be called modularity if the modules don't actually fit together?
The Space Shuttle Main Engine was handled in a different manner, top down, we might say. The engine was designed and put together all at once with relatively little detailed preliminary study of the material and components. Then when troubles are found in the bearings, turbine blades, coolant pipes, etc., it is more expensive and difficult to discover the causes and make changes. For example, cracks have been found in the turbine blades of the high pressure oxygen turbopump. Are they caused by flaws in the material, the effect of the oxygen atmosphere on the properties of the material, the thermal stresses of startup or shutdown, the vibration and stresses of steady running, or mainly at some resonance at certain speeds, etc.? How long can we run from crack initiation to crack failure, and how does this depend on power level? Using the completed engine as a test bed to resolve such questions is extremely expensive. One does not wish to lose an entire engine in order to find out where and how failure occurs. Yet, an accurate knowledge of this information is essential to acquire a confidence in the engine reliability in use. Without detailed understanding, confidence can not be attained. A further disadvantage of the top-down method is that, if an understanding of a fault is obtained, a simple fix, such as a new shape for the turbine housing, may be impossible to implement without a redesign of the entire engine.
Full report is here http://www.ralentz.com/old/space/feynman-report.html
I think part of the problem is that Boeing defined what components they needed, but not necessarily completely how they were to be built or function internally (for things that have internals). For example, they may have specified an electronic component by its inputs/outputs and working/environmental tolerances, but not anything about the internals. In theory, this "black box" approach should work pretty well, but - as we programmers know - side effects, edge conditions and unknowns are a bitch.
In the case of the Li-Ion battery and its external monitoring/charging equipment, different assumptions may have been made on either side. As for other components, sometimes one needs to know how they're to be assembled to make them correctly.
It must have been something you assimilated. . . .
Where your analogy breaks down is that weight matters for a plane. Plutonium doesn't make for a superior mousepad, but lithium ion has the advantage of a wider operating temperature range(specifically colder), high efficiency, power capacity, and energy density.
IE a lithium ion battery is going to generally be at least half the weight of any other chemistry for the necessary power/energy demands of the application, and you don't need to worry as much about heating them. For a plane, this makes you really want to use them.
Sure, you might only be saving 100 pounds for the battery pack, less than the added weight an extra obese passenger, but a couple hundred pounds for the batteries, a few hundred from the landing gear, engines, galley, cockpit, seats, wings, flaps, electrical systems, pumps, etc... Next thing you know, your plane is 10-20% lighter and that translates into extra speed, lower fuel cost, etc...
I don't read AC A human right
I know I'm late for a nice flamefest, but I'm sure someone will point out how Krugman was wrong about this when he discussed it almost two years ago: http://krugman.blogs.nytimes.com/2011/02/19/thank-you-boeing/
No, the problem is bureaucracy. Bureaucratic systems inefficient at management, partially due to a misunderstanding of and over-reliance upon digital data.
in other words: BAD MANAGEMENT
You attempted to give examples of "(IIRC) three" unspecified aerospace projects that had been prototyped using all-digital, proving nothing. Assuming your examples are real, absence of evidence is not evidence of absence. Aerospace prototyping failures that result in commercial product failures are a big fucking deal.
Look, here's what you are missing: When human lives are involved, prototyping things like...um...the F-22 or this 787 with a real-life prototype is *absolutely necessary*
The only way to conclude otherwise is to de-value human life. Do all the actuarial MBA-bullshit spreadsheets you want, you are still doing less work, less science on a complex project purely to save money *at the expense of lives*
Research 'military boondoggles' and you'll see Boeing has experience with this. The F-22 Raptor, be it digital-only prototype or not, is an example of the same **FAILURE OF MANAGEMENT** in all ways.
I'm not just talking 'operational' failure. There is a systemic problem and a systemic inability to do proper error correction...so it's like having clogged pipes that are also rusting.
That's the problem here. A fundamental flaw that with certainty will result in using bad metrics in the lab when deciding the statistical tolerances for the prototyping program....that's just one mistake.
Thank you Dave Raggett
Did they NOT have fires during the thousands of hours of test-flights?
These are my friends, See how they glisten. See this one shine, how he smiles in the light.
I'm calling troll....
I refuted your points, with specific examples using technical language (not 'buzzwords'). I gave the F-22 example. I used systems science and 'management science' terminology.
I offered a simple thesis...your approach is a cost-saving measure but ill-suited to situation like vehicles where human lives are at stake. I said your method sacrificed human lives for 'efficiency'.
That is a falsify-able statement you could have argued against...if you weren't a TROLL
Anyone reading this deep can judge but I deem thee a troll and you have rendered this part of the thread irrelevant.
Thank you Dave Raggett
prototyping also exists to do required tests.. If the FAA decides it's ok for you to fly with batteries with a known thermal runaway problem, provided you encase them inside some kind of fireproof enclosure, they generally want to see that enclosure tested to ensure it does what it's supposed to. And not just tested inside a computer.
tens of kg is important. Especially when you make literally thousands of design decisions. That 50 kg turns into millions of dollars over the life of the plane. 50kg saved on batteries means carrying 50kg more fuel or cargo. 50 kg of cargo is several hundred dollars/flight, and at hundreds of flights/yr, that adds up.
Designing a plane or a spacecraft or a car is an eternal battle against creeping mass.
under Carly Fiorina
Chair, President, CEO - BA, MBA (not a shred of actual engineering experience) President, CEO - BA, MBA (used to be a 727 mechanic . . . in the 70s) Senior VP - history major Senior VP - CIVIL engineering and a master's in engineering management . . . Dilbert says hello) CTO - BS/MA physics, PhD engineering I'm not arguing only aeronautical engineers could run an airplane manufacturer. That would be similar to saying only doctors should run hospitals/healthcare companies. BUT . . . if you accumulate too many 'profit' bean counters, an organization can often lose sight of the fact that it's the product that makes money not making money off the product. The only reason we get away with it in healthcare is because society pays us to do stuff . . . regardless of whether we do it well. "What's the difference between a doctor and a pilot?" A doctor isn't going to go down with you. -David Healy, MD Arguably the same could be said for senior management at most US corporations.