Why New Systems Fail

bfwebster writes "Over the last forty years, a small set of classic works on risks and pitfalls in software engineering and IT project management have been published and remained in print. The authors are well known, or should be: Gerry Weinberg, Fred Brooks, Ed Yourdon, Capers Jones, Stephen Flowers, Robert Glass, Tom DeMarco, Tim Lister, Steve McConnell, Steve Maguire, and so on. These books all focus largely on projects where actual software development is going on. A new book by Phil Simon, Why New Systems Fail, is likewise a risks-and-pitfalls book, but Simon covers largely uncharted territory for the genre: selection and implementation of enterprise-level, customizable, off-the-shelf (COTS) software packages, such as accounting systems, human resource systems, and enterprise resource planning (ERP) software. As such, Simon's book is not only useful, it is important." Read on for the rest of Bruce's thoughts on this book. Why New Systems Fail: Theory and Practice Collide author Phil Simon pages 251 publisher AuthorHouse, 2009 rating 8/10 reviewer Bruce F. Webster ISBN 9781-4389-4424-1 summary Risks and pitfalls of enterprise COTS projects Phil Simon has written a long-needed and long-overdue book. Most risks-and-pitfalls book in the IT category focus primarily on projects where actual software engineering is the principal activity. However, many of the large, expensive and often spectacular IT project failures over the past 20 years have little to do with software design and development. Instead, they involve a given organization selecting and implementing — or trying to implement — a commercial off-the-shelf (COTS) software package to replace existing legacy systems, either homegrown or also commercial. The reasons for such a move can be many: standardizing IT and data management across the enterprise, seeking new functionality, retiring systems that are no longer supported or supportable, and so on. By so doing, the firm (usually rightly) thinks to avoid the risks and expense of from-scratch custom software development. However, the firm (usually wrongly) thinks that such a project comprises nothing more than installing the software, training some users, converting some data, and turning a switch. A quick search on the terms "ERP" and "lawsuit" shows just how mistaken that idea can be.

Simon's book is far more informative and instructive than a Google search and should be required reading for all CIOs, IT project managers, and involved business managers prior to starting any such enterprise COTS project. He covers the complete lifecycle of such projects, starting with the typical expectations by upper management ("Fantasy World") and following it through system selection, implementation, and production, along with a final section on how to maximize the chances of success. Along the way, he uses several real-word case studies (with names changed), as well as a few hypothetical ones, to demonstrate just how such efforts go wrong.

What Simon writes is spot on. For roughly 15 years now, my primary professional focus has been on why IT projects fail. I do that both as a consultant (brought in to review troubled projects to get them back on track) and as a consulting or testifying expert (brought in to review troubled or failed projects now in litigation). I have reviewed hundreds of thousands of pages of project documentation and communication; I have likewise traced or reconstructed project histories for many major IT projects, including enterprise COTS projects. It's clear that Simon knows exactly what he's talking about and knows where all the bodies are buried.

The book itself is very readable. Simon's tone is conversational and a bit humorous; he occasionally dives into technicalities that would be lost on upper management, but always comes back to basic principles. The real-world and hypothetical case studies will have those of us who have been on such projects nodding our heads even as we occasionally wince or shudder. His coverage is exhaustive (and at times a bit exhausting), but his goal appears to be to give those managing and overseeing such projects the information they need to navigate the shoals. He goes into detail about COTS pitfalls such as project estimation, vendor selection, use of consultants, group responsibility, integration with legacy systems, data conversion, and report generation.

The first section of the book covers how and why firms decide to initiate a major COTS project. Besides the "Fantasy World" section that compares management expectations to what really happens, the book also covers why firms hold onto legacy systems, why they buy new (replacement) systems, and how they can (or should) make the decision among building a custom system, buying a COTS system, and "renting" enterprise software via a web-based software-as-a-service (SaaS) vendors such as Workday and Salesforce.

The second section covers COTS system selection. The book divides current ERP and COTS vendors into four different tiers based on company size and use (e.g., SAP, Oracle and BaaN are all Tier 1) and warns of the, ah, enthusiasm of vendor salespersons. (Old-but-still-timely joke: What's the difference between a used car salesman and a software salesman? The used car salesman knows how to use his own product and knows when he's lying.) The book then raises up front an issue often left (by customers) until much later: how will business processes change as a result of the COTS system we're acquiring? It then talks about selecting, if necessary, a consulting firm to help with the installation and project management.

The third section covers the actual COTS implementation process, including the overall strategy, roles and responsibilities, providing the necessary environments, data migration, testing, reports, and documentation. This section is a bit exhausting at times, but it is critical for exactly that reason: far too many firms launch into a major COTS acquisition without fully realizing just what it will take to get the system into production.

The fourth section briefly deals with life after implementation. In theory, one of the reasons a firm buys a COTS system is to avoid doing its own maintenance and support; the reality is that the firm often doesn't like paying those large annual maintenance fees and instead goes off on its own path, which is seldom a good idea.

The fifth and final section talks about how to maximize the chance of success in a large COTS implementation. This section builds upon the rest of the book, which has provided suggestions along the way. In particularly, it talks about how to deal with a troubled project mid-course in order to get it back on track.

Throughout the book, Simon puts a significant focus on human factors in project success and failure. He identifies issues such as internal politics, kingdom-building, reluctance to learn new systems, internal project sabotage, end-user resistance, and staff allocation. Simon divides firm personnel assigned to work on the COTS project into four groups — willing and able (WAA); willing but not able (WBNA); not willing but able (NWBA); and neither willing nor able (NWNA) — and talks about how each groups helps or hurts. Similarly, he identified four dangerous type of project managers: the Yes Man, the Micromanager, the Procrastinator, and the Know-It-All. Again, those of us who have been on major IT projects, particularly those involving COTS implementations, will recognize both sets of categorization and the risks they entail.

While Simon is himself a consultant, he is also quite frank about the role consultancies can play in COTS project failures. In particularly, he notes the tendency of consulting firms to underestimate project duration and cost in order to win business, as well as the frequent unwillingness to point out risks and pitfalls to the client, particularly if they represent something the client wants to do.

My few complaints with Why New Systems Fail are mostly production-related. Simon self-published the book; as such, the book's internal layout and graphic design leaves something to be desired. Likewise, his organization and prose could use a bit of editing in spots; he has a propensity for throwing in terms and abbreviations without clarification, and the technical level can vary within a given chapter. Almost all of his footnote references come from Wikipedia; his bibliography is small (just four books) and cites only Brooks from the cadre of authors listed above. None of this makes the book's content any less important or useful, but some of the very people who should be reading this book might well skip or skim it for those reasons. My understanding is that Simon is working on finding a publisher for the book, which will likely solve all those problems.

In the meantime, if you or someone you love is about to embark on an enterprise-level COTS project, get this book; I've added it to my own short-list of recommended readings in software engineering.

You can purchase Why New Systems Fail: Theory and Practice Collide from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.

Comments

Software Projects vs. Traditional Projects

[religious freak]

I was discussing with a friend how software projects are probably the most difficult to run and predict, especially with very large projects. He disagreed and said that all large projects are difficult - when you're building a bridge a multitude of things can and do go wrong.

That's obviously true, but how many bridges never get finished compared to the number of software projects that never get finished? It seems project management is very difficult for IT related stuff. So am I just being IT centric in thinking our projects are more difficult than most?

Re:Software Projects vs. Traditional Projects

[MightyYar]

how many bridges never get finished compared to the number of software projects that never get finished?

All bridges are essentially "open source". Plenty of bridges have failed, but the failures are right out there in the open, ready to be studied by anyone who wants to build another bridge.

In contrast, when a company's software project fails, the only people who learn from it are the ones involved with the project.

Re:Software Projects vs. Traditional Projects

[fuzzyfuzzyfungus]

Software certainly does have the disadvantage of being extremely complex(Both internally and, perhaps ultimately more serious, in its interfaces to other software and systems.) What gives it an extra edge of danger, I suspect, versus some other complex projects is the difficulty(particularly for those of limited technical understanding who happen to be involved) of intuitively grasping how the project is going.

You wouldn't want a non-engineer trying to micromanage bridge construction; but anybody can stick his head out the window and see how far across the water the bridge is today. The incipient cracks in the foundation might well be missed(as they often are), and the project can still easily go over time, or over budget; but it is hard(er) for fundamental delusions about progress to crop up.

A layman looking at a complex piece of software doesn't have nearly the same chances. A "nearly there" system with a few serious but non-systemic bugs looks like a broken unusable mess. A brittle demo being coaxed through by a guy who knows the system better than the back of his hand looks a lot like an intuitive interface. If your institutional structure or culture has any of the factors that encourage delusions, lying, yes-manning, or similar, the people who are supposed to have a grand vision of the project won't have the foggiest idea what is going on.

Didn't need a book to know this

[Em Emalb]

Not trying to be a jerk (hah, stupid buttface!) but the reason most new "systems" fail is for one of 4 reasons:

1) the decision maker(s) not understanding the actual requirements thereby causing a situation where they end up with a system that doesn't fit their needs

2) the third party or in-house developers not understanding the actual requirements thereby causing a situation where the system they've created either doesn't work or doesn't work as it should

3) the new system is too complicated/buggy/worthless and the end users of the system refuse to use it and/or complain constantly (I HATE CHANGE!)

4) all of the above.

There are more, but those are the big 3.

Re:Didn't need a book to know this

[MightyMartian]

I've had two projects fail ignominiously. One was my fault for not getting much more concrete requirements, and getting caught up in the "oh, and can you add this to it too?" The second was because I was basically lied to by a supplier who claimed their own product could do what it ultimately could not, and since it was a core feature of the system we were putting in place, the whole thing died, but not before consuming heap loads of money.

I learned a few things. The first is to get exact specifications. Let there be no wiggle room, no "well I thought it would do that" crapola. Extras can be added on once the core system is demonstrated to work, not before. Have a design philosophy and stick to it. As to lying suppliers, well, it's a lot easier to assess these things nowadays than when the one project failed in the mid-1990s. Still, I always keep in the back of mind "if software/library/whatever doesn't work, is there something that can".

Re:Didn't need a book to know this

[dave562]

As to lying suppliers, well, it's a lot easier to assess these things nowadays than when the one project failed in the mid-1990s.

What is different now from the 1990s? I've been involved with one software project that failed because the vendor promised functionality that they couldn't deliver. The client spent a significant amount of money on the project. Once it came out that the software couldn't do what the vendor promised it could do, the client sued the vendor and recouped all of their money plus legal fees. The client was able to sue because the vendor put it in writing.

Getting things in writing from the vendor is of paramount importance. Doing a needs analysis with the client before shopping around for software vendors is key. With a needs analysis in hand, you can present that to the vendors and ask them point blank whether or not their software fits the needs. If they say it does, make them sign a contract to back up their claims. Then they either deliver what they promised or they get sued.

Re:Didn't need a book to know this

[rgviza]

> I learned a few things. The first is to get exact specifications. Let there be no wiggle room, no "well I thought it would do that" crapola.

This is not realistic. You _can't_ get the requirements right up front because the users don't even know what they want until they have a system that doesn't have it. They think they told you what they want, but they didn't because they don't know themselves. A more realistic approach is to get the best requirements you can, and build enough time into the project to handle 1.5-2 years worth of scope creep because that's what's going to happen with any huge system.

If you try to hardline your users by forcing them into a corner with rigid up front requirements that they cannot possibly help you formulate, they'll simply go outside the company and work with someone who knows how to run a project better and you'll get laid off. (refer to Linus Torvald's rant about specs to see why specs and requirements done this way are useless, except as a starting point)

If you are prepared for scope creep, and frame the first release as an alpha, you will succeed. I've been doing this for 20 years and I've seen the approach you are talking about fail over and over even with PM's that have 30 years experience. They knew better but corporate policy forced them to operate this way. Inevitably the requirements were hopelessly incomplete and the users were pissed off when they had to sign off the project as complete because of what they agreed to, and in the end, the product did not meet their needs. The whole idea is to give the users the product they need. So even if you succeed in beating them on paper, and they are forced to sign off complete, you've failed.

Know what happens when you do this to your users? They hire contractors, who will be more flexible and give them what they want, and fire you. You are better off with a "Look this is a big system and it's going to take a while to get it right. Lets figure out what you think you need now, we'll build it, and use that as a starting point to flesh out your system."

XP for the win for corporate development, Waterfall = FAIL. Waterfall can only succeed if you are a software company building a boxed static product produced by someone that knows what they want.

At the end of the day a large corporate software project will take 10x longer than you think it will. I've never seen one that didn't. I've seen plenty that failed. Plan accordingly.

-Viz

Tolstoy's version

[T.E.D.]

People have written oodles of books on this subject, because there are oodles of different ways to screw up a project.

The best insight on this subject comes from Tolstoy, not Brooks. He was talking about families being functional, not software, but the principle is the same.

All happy families are alike; every unhappy family is unhappy in its own way.

A far better method of approaching this issue is to study projects that don't fail, not ones that do.

Underpromise and over-deliver! That's my motto...

[filesiteguy]

...or at least one of them. I haven't read the book yet - but it is now listed as a to-do in my list of to-do items taking up space on my blackberry.

In any case, I'd be curious what the answer is. In my short software development experience - only since '93 have I been doing enterprise-level development - I see one factor being the overwhelming key to failure.

Communication.

When you have analysts and developers (who are notorious for not being communicative in the first place) trying to interface with executives and managers (who are trying to CYA) then you have a perfect storm brewing.

Add to it, the fact that COTS solutions rarely actually fit the needs of everyone, and you subscribe to failure. A classic example that I just saw this week is with the California State Child Welfare lien processing system, written by Accenture. I asked for a minor change in the file layout some months ago. Only this week did I hear that I'd need a change request and that they'd get back to me in a few months. :P

By contrast, I've written my own in-house custom software systems for the enterprise. (One system in production has well over 500 concurrent users on any of fifteen different modules.) When teh customer(s) request a change, then it can - depending on complexity - be implemented and tested in a matter of days. Of course, harping on the communication theme, I'm in constant communication with teh customer, the end-user (if different), my developers, and my analysts. (I'm a PHB in the middle.) I make sure that we under-promise and over-delivery whenever possible.

The bigger the rollout, the harder the crash

[petes_PoV]

"But we don't have time for a pilot"

Also heard as "Why, don't you have confidence in your project"

Putting aside the sheer commonsense approach of not giving a porsche to a newly passed driver, most projects are run in a state of panic. Panic that the timetable is slipping (although this is almost always due to poor time-estimating, it seems to get presented as being due to slothful or untalented techies), Panic that it's costing too much - again due to poor cost estimation, rather than ovespending. Panic about bugs, Panic about training (ha!). Panic about compatibility with other systems. Panic about all the little patches, workarounds, working practices and hacks that have developed in the old system - that everyone knows about, but have never been documented.

All these, could have been identified and most of them fixed just by running a small scale prototype in parallel to the existing system. However by the time the project is halfway through, most of the directors are firmly engaged in either "buyers remorse" or utter denial. They become deaf to bad news and generally take full aim at the messenger, while leaving the culprits of all the problems unscathed. This is usually because all the biggest mistakes are made right at the start - in the design stages. However, these have been completed and signed off, so by definition cannot be at fault. The blame gets transferred down the line, to the people who have their hands-on right at the time the deadline is due. It's the original smoking gun: "The project ran over time / budget today - you were working on it when that happened, therefore you must be to blame". It's simplistic, always wrong and always starts off the finger pointing part of the process. You can't get away from it.

Although the biggest problem I see is "seagull" consultants. They fly in, make a lot of noise, crap over everything and fly off. The trouble usually only surfaces once they've disappeared.