Lean Software Development

Jim Holmes writes "Mary and Tom Poppendieck's Lean Software Development: An Agile Toolkit is a great read for anyone interested in agile software development. That includes developers, leads, and managers interested in speeding up development cycles, improving quality, and getting their customers the best value. This book's been out since May, 2003, but it's well worth picking up. The concepts within are absolutely applicable now, and will continue to be for quite a few years." Read on for the rest of Holmes' review. Lean Software Development: An Agile Toolkit author Mary and Tom Poppendieck pages 240 publisher Addison Wesley rating 9 reviewer Jim Holmes ISBN 0321150783 summary Toolkit for getting agile development in your organization.

Lean Software Development is full of pertinent comparisons between the current state of software development and the massive changes in manufacturing over the last three decades, specifically demonstrated by the Toyota Production System, and 3M's innovative atmosphere for bringing products to life. The Poppendiecks make a great case as to how similar changes in software development can reap great benefits in the software production industry. Who It's For The book's very useful for anyone involved in or around the software development process: developers, leads, managers, and corner-office types. Corner-office types won't get as much out of the book as those in the trenches, but the Poppendiecks' arguments against overly-constraining process management systems may help high-level managers come to understand that such systems can actually hurt production.

Who It's Not For This book isn't for closed-minded folks who think the waterfall method and a preponderance of documentation and process control are the bee's knees. The book talks specifically about how Six Sigma, Capability Maturity Model (CMM), Capability Maturity Model Integration (CMMI), and Project Management Institute (PMI) certification can drag down development productivity and quality. Also, it's not for folks who are unwilling to consider that shorter delivery cycles improve feedback, quality, and lower cost.

(Note that the authors specifically point out that agile development does not mean tossing out all documentation and process.) What It Covers The book is labeled a "toolkit" for lean development, and it describes 22 "tools" -- that is, approaches which will help an organization move to a leaner development system. The authors start off with a great explanation of what lean practices are and how they can benefit software development. They then move on to more detailed coverage of important principles.

The book's broken into chapters covering the seven principles the Poppendiecks lay out as fundamental to agile practices: eliminating waste, amplifying learning, deciding as late as possible, delivering as fast as possible, empowering the team, building in integrity, and seeing the whole. Those seven principles may sound like marketing blabberspeak, but the Poppendiecks nail each section down with terrific discussions of applicability.

They've also got great examples tying the principles into how manufacturing has so drastically improved its processes. Each chapter concludes with a "Try This" section aimed at getting your group moving in a lean direction.

The second biggest benefit after the book's content is the extensive reference list. There's an impressive bibliography, and each chapter is loaded with footnotes referencing various books, articles, etc. This gives interested folks a great guide for further reading.

The book's summary chapter is especially good. It concisely wraps up the book in the somewhat tongue-in-cheek format of an instruction sheet for the tools the Poppendiecks have laid out. The "Caution - Use Only As Directed" section is particularly useful because it shows how one should not use the principles: "Eliminate waste does not mean throw away all documentation," and "Deliver as fast as possible does not mean rush and do sloppy work." The summary also breaks out high-level details for implementing in large and small companies. The authors are particularly helpful in pointing out strategies for dealing with difficult process improvement programs such as Six Sigma, CMM, and/or CMMI. They point out the political aspect of how to approach implementing agile methodologies in organizations constrained by such "helpful" policy systems.

There's also a note for folks working in safety-related fields where regulations and immense processes dictate how to do work: Shortening cycles in such environments can better ensure people aren't killed by software failure. What It Doesn't Cover Despite the great coverage of the principles and tools, this book isn't a detailed guide for implementing agile processes at your organization. The authors are very adamant that no two organizations function alike. Implementing agile processes requires some careful forethought before jumping in. The authors don't advocate any one methodology over another, so don't look to this book for help in deciding whether you want XP, FDD, SCRUM, or any one of the other alphabet-soup-of-the-day agile buzzwords.

Additionally, I thought a few items were given pretty cursory coverage. One example is in the chapter on late decisions where the authors breeze right over implementing a quick persistence layer to put off deciding on exact database implementation. I particularly would have liked more detail in that item. On the flip side of that; however, is the great detail given to value stream mapping, feature implementation burn rates, and several other very, very useful items - so my complaint is really that one particular item I'm working on right now wasn't covered as well as I'd have liked. Bottom Line This really is an important addition to your reading list if you're at all interested in learning how an agile environment can increase your speed, quality, and cost effectiveness. It's a great book if you're in need of guidance on how to look at and improve your current environment. It's also a great book if you need backup for convincing either your co-workers or management that a move to agile is necessary.

You can purchase Lean Software Development: An Agile Toolkit from bn.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.

Comments

hmmm

[Shut the fuck up!]

What an unfortunate last name the authors have.

eXtreme Programming?

[Karma_fucker_sucker]

What's this? A repackaging of eXtreme Programming?

I'll have to write a book myself: "Anorexic Programming" by Smart Ass. "The Ultimate in Lean and eXtreme Programming"!

Re:eXtreme Programming?

[dubl-u]

There are a lot of similarities to XP.

And there's no competition there. The Poppendiecks led several sessions at Agile 2005, the big XP conference in the US. XP is much more about what goes on at the team level, while the Poppeniecks are more interested in broader business and corporate culture issues.

Close-minded persons

[lilrowdy18]

I think the waterfall method and a preponderance of documentation and process control are the bee's knees.

Seriously, who uses the phrase "bee's knees" anyway and what is so great about bee's knees?

Re:Close-minded persons

[ch-chuck]

A bee's "corbiculae", or pollen-baskets, are located on its tibiae (midsegments of its legs). The phrase "the bee's knees", meaning "the height of excellence", became popular in the U.S. in the 1920s, along with "the cat's whiskers" (possibly from the use of these in radio crystal sets), "the cat's pajamas" (pyjamas were still new enough to be daring), and similar phrases which made less sense and didn't endure: "the eel's ankle", "the elephant's instep", "the snake's hip". Stories in circulation about the phrase's origin include: "b's and e's", short for "be-alls and end-alls"; and a corruption of "business".

Re:Close-minded persons

[Anonymous Crowhead]

Ah, the twenties... "Give me five bees for a quarter", we used to say. Now, about that onion strapped to my belt...

Yeah right.

[BluedemonX]

RE: "Eliminate waste does not mean throw away all documentation,"

Cobblers!

I remember distinctly reading on some Agile XP whatever site that CRC cards (the documentation is the code and unit tests!) are used long enough to get the devs on board with what to do AND THEN THEY ARE DISCARDED.

Stuff's real good if you're doing your comp sci 101 homework but in the real world you need a process.

Re:Yeah right.

[ChaseTec]

CRC cards are not documentation. They are an artifact of the development process. You can hang onto them for historical records if you find them useful, but they are inherently out of synch with the final product (except by coincidence).

Remove the mention of CRC cards and you've perfectly described documentation.

Re:Yeah right.

[dubl-u]

I remember distinctly reading on some Agile XP whatever site that CRC cards (the documentation is the code and unit tests!) are used long enough to get the devs on board with what to do AND THEN THEY ARE DISCARDED.

Yeah, people often freak out about that. I tell them, "Ok, then you can keep the cards." They will happily put them in a box and then never look at them again. Myself, I don't use CRC cards much, I just sketch UML on the whiteboard. I do tend to leave the last few months of story cards up on the wall, though.

Note that the most important documentation on an XP project is the acceptance tests, which you can think of as either machine-verifiable specs or automated versions of what QA people often do manually. Those say what the product is supposed to do. One framework for this is FIT, which uses specially structured tables in HTML documents.

Unit tests, on the other hand, are where developers say what individual chunks of the code are supposed to do.

Seems to me like it's an oxymoron...

[GecKo213]

This book isn't for closed-minded folks who think the waterfall method and a preponderance of documentation and process control are the bee's knees.

Most upper management are only about those things! Besides, having coded for companies before, I know that if you don't properly document your code and make sure you have a preponderance for process control in place typically the whole thing goes to shit. And what is this about the bee's knees!?

Re:Seems to me like it's an oxymoron...

[kaoshin]

"the issue is one of understanding, not of documentation, therefore you should not overrate the value of documentation. Your goal is to ensure that maintenance developers understand how the system works so they can evolve it over time, not to produce a mound of documentation that they may or may not use."

-Taken from this essay on agile documentation

I agree with the above, but it is my experience that the reinforcement on developers generally needs to be in creating more documentation. The environment will naturally make all the difference. In the nasty corporate arena (my habitat), many people feel that being the only one who knows something, is their ticket to job security. As a result, they will not comment code or divulge information to anyone, etc. Unfortunately, management encourages these insecure people through among other things, a lack of employee loyalty and an eagerness to cut costs by giving valuable and veteran employees the axe, among other nasty things. Thats my take anyway.

Bed goes up. Bed goes down. - Homer Simpson

Re:Seems to me like it's an oxymoron...

[dubl-u]

I agree with the above, but it is my experience that the reinforcement on developers generally needs to be in creating more documentation. The environment will naturally make all the difference. In the nasty corporate arena (my habitat), many people feel that being the only one who knows something, is their ticket to job security. As a result, they will not comment code or divulge information to anyone, etc.

Ask yourself what goals you're trying to achieve with documentation. It sounds like you care most about knowledge transfer between developers. Agile methods solve the same problems, but with different methods. Knowledge transfer, for example, are solved with practices like pair programming, collective code ownership, acceptance tests, unit tests, and putting the team in a war room.

Of course, the people steering the project can order more documentation at any time. On one recent project, we handed the code off to a team at another site. In addition to having a bit of a developer exchange program, we also took some time in the last couple of weeks to write high-level architectural docs. By doing just-in-time delivery on the docs, we saved a lot of effort keeping docs in sync with the code. And because it was scheduled as part of the regular work stream, we didn't rush; it was just another chunk of work.

Lean Six Sigma?

[notdanielp]

Is this a derivative of the methodologies behind Lean Six Sigma?

There's big money in that, my graduate software engineering course last semester had a speaker in from a NASA contractor that pushed LSS as a way to manage all kinds of different engineering and production variances e.g. misfiring rocket engines.

Re:Lean Six Sigma?

[etedronai]

I don't think that six sigma and agile software development are derivatives of each other. Agile Software Development is actually meant to attack a different kind of problem than Six Sigma, even Lean Six Sigma, is meant to attack. The idea behind Six Sigma is that you can get more efficiency out of a process by always doing things exactly the same way. Try to cut down on the variance as much as possible. This is why it does well with production line/engineering type activities.

The idea behind agile software development is that you can not apply production line type ideas to software development because two software development projects are never the same. This is why estimating and planning for them is so difficult. Agile software development says that you should just admit that developing a software product is more like research and less like running a production line and plan it accordingly. This means doing short explorative iterations that slowly build up to a larger deliverable and constantly inspecting the process and the schedule and making course corrections on the schedule.

Bees Knees?

[Shadow Wrought]

Banana oil, I say!

Just went thru this

[bobalu]

We just went through this with a new project. One of the usual suspects started putting out huge emails with a hundred questions/requirements/concerns such that you'd need to solve desktop fusion before protyping the thing and the lead architect just basically slapped him and said no, I'm not even going to answer this - let's keep it informal, get it going first and see what we can do and what makes sense. Probably saved the company $100k that day.

Doesn't work for everything, but when it does, use it.

Re:Just went thru this

[RingDev]

And will cost $500k in the long run. Those questions, concerns, requirements, etc should have all been recorded. The future user who just got smacked down for expressing his/her concerns will likely develop a less then positive view of the IT department, and be significantly less likely to partake in future design meetings where those requirements and concerns will become much more imporant.

I have been dragged through numerous develop now, design later! projects, and each one of them has gone over schedule. I'm still trying to clean up this current project that I got hired into. The users were still bringing us new requirements for the invoicing system 2 months after it went live. Not to mention the actual framework that the app was built off of was a hodge podge of misc code. Finally we are getting things standardized and working on a solidified portal and framework system. Poor planing, a complete lack of requirements gathering, a complete absence of a clear unified design, and many other poor initial lead and management decisions have us still working on corrective maintenance for an app that was released six months ago. All for the sake of developing and releasing the first version in 4 months. No thanks, I'd take 4 months of design and huge emails up front in exchange of 8 months corrective maintenance any day.

-Rick

Re:Just went thru this

[fupeg]

You guys are both wrong. It's not the method that matters, it's the people. Good software engineering is the key in either system, and neither system is perfect for all situations.

The key to agile programming is being able to actually handle massive requirement changes. I'm talking about the kind of requirement changes where all of those precious unit tests and acceptance tests you wrote get invalidated overnight. If your requirement changes are more minimal, then they're not going to be a problem with either method. They key to being able to handle massive requirements changes is good software design. Excessive coupling between components means that when one component becomes worthless (because of said changes) its hard to salvage other components that are still valuable because they are too tied to the deprecated component. Without good software design, agile programming won't help in the worst case scenario. This is especially important for agile programming, because its approach increases the likelihood of the worst case scenario.

Now I will say that a waterfall approach leads to several bad practices. First, it leads to a mythical man-month type of fallacy. Management tends to think that because of all the documentation present that they can manipulate "resources" like pawns in a game of chess. Second, it encourages bad design because you think you know everything. Third, and this is the worst to me, it gives false value to low value assets, i.e. people who don't actually produce anything. When a company values its process more than its employees, then it winds up hiring lots of people whose only function is to manage the process. Of course this is not unique to waterfall companies, but their emphasis on process does encourage this.

Agile programming also encourages several bad things. It encourages the over-valuing of development and under-valuing of design. I've often seen programmers with the attitude that's ok to build something that sucks because they will make it better in the future. Second, it tends to encourage overly-conservative programming. Its faster cycles discourages tasks that don't easily fit into these shorter cycles.

Of course the biggest limitation of agile programming comes from its roots. It clearly shows its consulting roots with its need for customer involvement. The kind of customer involvement it wants is very expensive for the customer (if we're talking enterprise software, maybe it's cheaper for consumer software.) If the customer is already paying you to build the software, then they might be willing to make this investment. If instead the customer is only a potential customer who might buy the software when it's done, then they are much less likely to make such a large investment.

The Winds of Change

[OneOfThree]

This book's been out since May, 2003, but it's well worth picking up. The concepts within are absolutely applicable now, and will continue to be for quite a few years.

Is it just me, or are things changing too fast when a development process has a shelf life of less than two years?

If you want to feel your head really spinning, pick up a copy of The Mythical-Man Month. Things don't actually change that much.

Re:The Winds of Change

[TrappedByMyself]

It's not so much that development processes are coming and going, it's just that people are tailoring the processes around what works.

I've been seeing this agile stuff for almost 8 years, and that doesn't mean it wasn't around before that.

So, gather around boys and girls, here is how you write software (does not apply to stuff like mission critical embedded software)

1. Ask customer what they want
2. Build something
3. Show it to customer, and ask what they want changed

If you make that cycle short, have good engineers, reasonable customers, and competent management, you will rule the universe.

What happens is that projects often have stupid and/or lazy people involved, so there are tons of failed projects. So, awhile back, the academics get together and come up with this deal where you do this extravagent design/requirements process upfront. TEH SAVIOR!!! ...managers rejoice...projects continue to fail...however, the projects with good people continue to prosper. So, what is wrong? oh, we need agile, iterative, incremental, eXtreme, [insert buzzword here] processes. TEH SAVIOR!!!...managers rejoice...projects continue to fail...however, the projects with good people continue to prosper. Things are a little better though, because the processes are closer to how good people do things.

Requirements?

[I_Want_This_ID]

I don't have a problem with most of these development methodologies perse, but most of them seem to lack the entire concept of DATA and INFORMATION.

I haven't read the theoreticals of a lot of these methodologies but have worked in several places that say they practice them. What I've found is a bunch of developers (not saying anything about their programming skills) that don't understand databases or design of data structures. I find it difficult to extend many of these systems that are frankly poorly designed. Do these methodologies include some prep work on gathering business requirements and understanding the underlying information relationships?

It's About Time

[MikeyTheK]

One of the least considered, studied, and written about things is project methedology. It seems like all the books about the subject relate to automotive production, where the Japanese flooded the market with their philosophy. Afterward, automotive companies hoisted these philosophies on unsuspecting suppliers. The number of paradigm shifts that auto suppliers have endured as a result over the last 20 years is amazing.

Yet I have never had a large client try to push a project development methedology for software on us. There has never been discussion of any of these, and it seemed that for the most part, we as a group were on our own when it came to evaluating various options.

I've been waiting for someone to start talking about the various paradigms and structuring philosophies for a while. For projects as complicated as large software systems, it is frequently a lot of trial and error, followed by more of the same. Finding any comprehensive discussion has been impossible. Hopefully this will foster more books on the topic. Software is becomming such a critical part of corporations, government, and daily life, that not only are the tools that we use important, but so is the way that we use them.

I would also be interested in seeing some sort of discussion as to how various team-structuring philosopies fit with various classes of development tools. For example, database development lends itself to a different approach than web design, simply because of the design process, hurdles that have to be dealt with, degree of expertise, and expense of the individuals that participate in the project.

Re:It's About Time

[mseabury]

I work for a 3 year old software startup. We have just been audited 3 times by customers or portential
customers for our product.

Even though they haven't pushed their software design process onto us, they have all made it clear that there must be processes to trace the requirements through developement, testing and releasing of our software.

CMMI and its use(ful/less)ness

[xenomouse]

The review mentioned CMMI, so i figured i could share my personal knowledge of it to give others at least a basic feel for it. A few years back i worked for a defense contractor at a military base. Don't get worked up: we were glorified librarians, only occasionally writing internal apps to make lights blink, pocket strings loosen, and other such things occur. Our group (being the smallest one on the base), got the crappiest contracts. One of them was to ensure that all software development done on the base was CMMI level 3 compliant. I was one of the "lucky" ones sent to the CMMI classes to be indoctrinated. CMMI has a number of levels, starting at level 1 (very little bureaucracy, minimal documentation, and barely organized) and going level 5 (an amount of bureaucracy that would put Brazil ["Mistakes? We don't make mistakes."] to shame). Working at level 3 is right in the middle of the spectrum, with just a bit too much bureaucracy for my tastes (when you need to generate a paper trail just to write a simple function, it's a tad too much). There was a rumor going around that there was one (maybe two) companies that qualified as CMMI level 5, and they were believed to be in india (where the cost of maintaining such a level would not be prohibitive to sustain). Of course, that may have changed in the past few years.

Now comes the question... why would anyone want such a level of bureaucracy? Well, in our case, we were responsible to the government, and from what i can tell, the government equates "paper trail" with "accountability and transparency." In other cases (commercial software), this would allow a company to switch contractors if the current contractor started acting nutsy or broke a contract in some way.

Think of the group paying for the contract as "the boss" and two contract groups called "Bob" and "Tim." The boss wants to pay someone to develop the MegaApp for him. Bob and Tim both make promises about how fast they can develop the MegaApp and how much it will cost and how much quality the MegaApp will contain after they work their respective voodoos. Bob makes some egregious claims, and the boss (because he's a manager) believes Bob and passes over Tim's more realistic promises. Bob, of course, fails to deliver on his promises, but the boss saw that some of what Bob did met some of the requirements. Since the boss made sure that Bob and Tim implemented CMMI level 3 before they could even be considered for contract, he has the option (or believes he has the option) to take the deliverables (including CMMI documents) to Tim. Tim, who is accustomed to using CMMI level 3, should theoretically be able to pick up the project in a short period of time and run with it due to the high level of documentation.

Just like communism, it can sound good in theory and look good on paper but will probably only work well in a perfect world.

Note: Carnegie Mellon developed CMMI.

Disclaimers:

• The CMMI classes i attended were by no means comprehensive.
• CMMI has been updated since i took the classes.
• There are several CMMI models, each with their own purpose.
• I am naturally predisposed against CMMI.

Re:CMMI and its use(ful/less)ness

[Anonymous Coward]

You're missing the whole point of the Capability Maturity Model when you state this:

"Working at level 3 is right in the middle of the spectrum, with just a bit too much bureaucracy for my tastes (when you need to generate a paper trail just to write a simple function, it's a tad too much)."

Even though you may understand your function as it is written now, when you look back at your code or someone else does in the future, having the documentation of what the function does along with when and why it was created is important.

Saw a presentation of this is Redmond

[bADlOGIN]

At the borg HQ even. Lean basicly packages agile concepts into a business-tasty format. There's nothing in Lean that contradicts with XP or Scrum (in fact, it tends to amplify it). The fact that these practices come from proven manufacturing and distribution practices give the concepts more credit . If you're having trouble "selling" either XP or Scrum, take a look at this. It should sell itself.

Agile is not for commercial software development!

[Anonymous Coward]

We had Jim Highsmith, Mr. Agile Manifesto, talk to our engineering team. What a waste of money.

If you believe Agile works, you probably also believe Google can be implemented with JEEE and Oracle.

These Agile guys really don't know what it takes to release large commercial products with million lines code, with many dependencies, many languages, requiring marketing campaigns, press tours, support training, etc.

More versions cost more money.

Sure Agile might work for small internal IT projects where you have one customer and you can roll out changes whenever you want without any change control or worries about upgrades or localization.

Agile is for "lean" projects only.

Re:Yep. Anyone who hasn't read F.P. Brooks...

[dubl-u]

There's nothing fundamental in Lean, XP, or Scrum that Brooks didn't manage to say 30 years ago(yes. 30. 3-0. Thirty. Near 1/3 of a decade. check the copyright).

I don't think that's entirely true. Test-driven development, for example, seems pretty new. Hardware cost had to be pretty low before each developer could compile and run all the project tests every few minutes of development. I also don't remember Brooks talking about weekly iterations.

That's not to knock "The Mythical Man Month", by the way. I completely agree that it's required reading.

Re:Agile is not for commercial software developmen

[dubl-u]

If you believe Agile works, you probably also believe Google can be implemented with JEEE and Oracle.

That's boldly, if unintentionally, ironic. Not only is Google hiring all the Agile developers they can find, but many agilistas have a lot of contempt for the ultra-heavyweight EJB-style approach to things.

These Agile guys really don't know what it takes to release large commercial products with million lines code, with many dependencies, many languages, requiring marketing campaigns, press tours, support training, etc.

There's no question that complicated projects have to be done differently than simple projects. But even there, you can place approaches along an agile/non-agile spectrum. You also shouldn't mistake "I don't know how" for "it's impossible".

Becoming agile also requires a fair bit of supporting infrastructure. For example, my Extreme-Programming-built code bases typically have a 1:1 production-to-test-code ratio. Bug rates for many XP projects are well under one per developer-month. Quality at that level enables fantastic agility in ways that seem impossible at typical quality levels. If you regularly have production relases with zero bugs found, weekly releases don't seem nearly as scary.

space shuttle software is CMM Level 5

[MarkEst1973]

They write the right stuff

The software in the space shuttle cannot fail. billions of dollars and people's lives are on the line, so it cannot fail.

At CMM level 5, you don't fix the bug when you find it. You fix the process that let the bug happen in the first place.

CMM Level 1 is no process. anything goes, really.

CMM Level 2 is fixing the bug and documenting that you found it. That more or less boils down to using a bug tracking system, keeping good version control, and everyone following this process.

CMM Level 3 is the software engineering level. that basically means that everyone in your organization builds their software similarly (software design and documented use of design patterns is good here).

Levels 4 and 5 is all about keeping a database of your processes and fixing your processes. Process flaws cause bugs, not code errors. You don't fix your code, you fix your process.

It's important to note the budget for the team writing the space shuttle code. Lots and lots of money and lots and lots of time go into that software. Ordinary application writers like us don't get that luxury.