Slashdot Mirror
Anatomy of a Runaway Project
JCWDenton recommends a piece by Bruce Webster revealing some insights into a failed multi-million-dollar IT project. "The following document is the actual text — carefully redacted — of a memo I wrote some time back after performing an IT project review; names and identifying concepts have been changed to preserve confidentiality (and protect the guilty). The project in question was a major IT re-engineering effort for a mission-critical system; at the time I did this review, the project had been going on for several years and had cost millions of dollars; it would eventually be canceled and the work products abandoned. The memo itself provides an interesting glimpse into just how a major IT project can go so far off the tracks that nothing useful is ever delivered."
It's sort of interesting, in a vague way, but you can read much more dire and funny stories on (the highly recommended) the daily WTF. My favourites would have to be the hotel reservation system from hell, the story of VirtuDyne and the digital donkey and a case of the MUMPS.
I don't see a similarity, really.
Wine is actually an example of something extremely rare - a project that looked like it was doomed from the beginning, took millenia to get to the current state, but achieved usefulness anyway. Most of the time it works and when it doesn't, most of the time it's just common bugs, not incompleteness.
This is Slashdot. Common sense is futile. You will be modded down.
Damn. That's the exact phrase I've been looking for. I don't know how many times I've seen hard truths and unpleasant realities float up the organization and stop dead about 3 management levels below where someone could do something about it. Just like sonar, the people "listening" above the thermocline will never hear anything occurring below it.
Welcome to the Panopticon. Used to be a prison, now it's your home.
I'm sure that there are other similar companies out there, but much of the language and all of the circumstances seem very familiar. Just curious, but how many other companies use the term "deliverables"? IBM, after purchasing Rational Software, decided that it was a good idea to move all projects to this process. About 2003, there was a huge stir within the company to document everything into a "process" and wasted months (nay, years?) in fluff process documentation that yielded no benefit. It is very interesting that their stock is doing so well at the moment. Lots of folks are jumping ship like mad right now . . .
StyleChief
Strange women lying in ponds distributing swords is no basis for a system of government! -M. Python
ObOwnExperience: One time, I had to do some maintenance work on a very large piece of badly-written and cruft-ridden code, ended up rewriting large tracts of it, reduced its source size by an order of magnitude and the binary size by three orders of magnitude. Also found some buffer overflow Heisenbugs which the previous maintenance guys had known about but bypassed by padding the object files. There's something... bothersome about corrupting a file in order to make a bug not be visible.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
"ISSUE: There isn't enough intellectual honesty within the FUBAR project. Managers reject or explain away bad news and real problems, looking instead for people who will tell them what they want to hear. "
This has easily been the #1 reason I have personally witnessed for project failure. I am in the process of witnessing it right now, even, with what seems like a relatively simple project. The suits and supervisors along the way are either not responding to requests for information, or change their request for features.
Bearded Dragon
Could have been any large company I worked for too in the late 90's early 2000's.
The last big company I worked for like that had the same memo written up about the project I was working on. It was reviewed by management and they interviewed 100+ developers on the project the following week.
It was concluded that the report was right. The next day, the management called a 10am meeting to tell us that after 10 years the project had been canceled. March 31, 2001 more than 300 people were given 2 hours to clean out their desk and leave the building.
People took everything that wasn't nailed down.
A year later after checking up on the project and finding out that there was a release less than 6 months after the mass layoff that the project had been outsourced to India.
Readers of the British magazine Private Eye might get the ideal that BigFirm was British Telecom (BT) and that the project might have something to do with centralised medical records. However, such a suggestion would be entirely misleading and completely wrong. -- Strobes
On a project a few years ago (your typical Fortune 500 $LARGE_COMPANY here), our PM was forced to declare that a large release that had taken 6 months to get to "it's kinda working" was "complete" and shipped to UAT even though system testing was incomplete and all we did was give the end users a pile of steaming shit. But the director of the group got to "meet" his deadline, and therefore get his much-desire performance rating.
Of course fixing bugs once the app is in UAT is four times as difficult as in the integration environment, with the corresponding lag in defect correction time. So UAT went on and on and on... until it was supposed to be the final release date. Said users were hysterical and pissed off, and said director was out of his fucking mind trying to come up with inventive ways to ship said steaming pile of shit to production while blaming someone else for the smell of said pile.
His first executive action was to fire the PM and have him escorted out of the building (he was a contractor like me). His second action was to request that the Offshore Solutions team add four more developers to the already swelled-beyond-comprehension development team in India. You know, throwing bodies at the problem. The next thing was to go to his VP and claim that he had been misled by the PM, who by this time was checking out the classifieds at home and therefore unavailable for comment.
In the end, we all went through the usual death march and shipped the thing about three weeks late. The director got his rating (not his fault you understand) and the users had to deal with the remnants of the steaming pile of shit. I get paid either way so no skin off my ass.
Projects are late (or they fail) because the people who are supposed to be in charge of delivering them have no fucking clue as to how software is developed. Fix that problem, and you'll ship all the software you want on time and under budget.
I'm fortunate to be in a project right now where the guy in charge is a former developer who doesn't require a bonus to pay his mortgage, and all the two PMs do is move little bars on an MS Project file while mercifully leaving me and my team alone to actually write the software. We've released two major versions of the app so far the past two years and are on track to deliver the third version sometime this October. On time and under budget. The secret? Iterative development (SCRUM-like) with heavy user involvement in feature sprints. No waterfall bullshit for me anymore, thank $DEITY.
Web2.0: I love when people Flickr my cuil and digg my boingboing until my google is reddit and I start to yahoo
Wow.
It's scary how much this resembles the ongoing debacle in the company I currently work for, as our "new" workflow application sucks millions of dollars into a black hole (and will continue to do so for the same reasons as the article talks about).
Everyone who actually has to use it, hates it. None of the developers has ever talked to any of the end users. It is fundamentally broken in design (most end users are on a WAN or the internet, but it is designed to be used on a high-speed, low latency network with functioning QoS). The sheer amount of FAIL in this project is staggering. Yet it continues to have money poured into it, all because it is the CEO's son's idea.
Meanwhile, the "old" workflow - which made us market leaders - has been left to languish and subsequently the competitive advantages it gave us have all but disappeared (we used to have technology years ahead of any competitors - but that was years ago). The true tragedy is we had plenty of smart people who not only explained why the project was doomed to failure, but offered (and in some cases implemented, only to be slapped down) functional and viable ideas to improve the existing tools. I expect all these people will have finally given up and left by the end of this year (as I probably will).
(Of course, this is far from the only issue causing our current drain-circling, but it certainly isn't helping, and it's happening for the same reason our other problems are - nepotism and incompetent management.)
We've all gotten burned on projects that got out of hand, but I often wonder why it happens, over and over. Hubris?
I've seen projects where the requirements document was 1000 pages and growing exponentially. For an email server. I remember one project where it didn't matter if code even compiled: we had to ship what we had, because we couldn't delay any further. I made the mistake of expressing my views to the wrong people on that one, and was told in no uncertain terms to shut up if I wanted to continue working there.
I've seen more than one project fall flat on its face because the original requirements were wrong, like trying to develop PC software for an industry that was 100% Mac.
...laura
. It reeks of "amateur", and would be ripped apart by just about any manager it was given to.
I have a high degree of confidence that many managers could probably think they were ripping it apart, but my guess is your average slashdot poster (let alone your average decent engineer) could probably respond competently to each charge, were said manager competent enough to have the responses you gave be real commentary rather than contrary rhetoric.
Manager: Please outline the facets of this project that can be eliminated.
"This is part of the problem. The project has been so poorly organized and tracked that no one has a current of outline it. It's possible that we *can't* outline it."
Manager: What makes this coder more qualified than the coder who wrote it?
"As you'll see I mentioned, the developer reviewing the code was aware of widely-known good practices in development -- such as the use of well-named constants, rather than 'magic numbers.' The coder writing the code was either unaware of these or unwilling to apply them, which quite likely means he's less qualified."
Manager: What the fuck does [fragile] mean?
"It means that adding new features without breaking existing ones is likely to be difficult. The poor organization makes it easy to accidentally step on something important to who-knows-how-many other places in the code. It also means the application doesn't have broad ability to handle the set of all possible inputs robustly -- there are enough cases that aren't anticipated in the code (or may not be anticipated -- it's hard to tell with the poor oranization) that it will likely crash regularly."
Manager: This guy is one of those "I can do it better using the new language i learned in college" kids
*rolls eyes* -- the idea that a manager who'd say this is common is a complete flight of fancy. Where the hell are you going to find a manager in corporate America who is hostile to Java and sees it as something "new" that's the province of green college kids?
Manager: This guy is obviously in over his head and fears his job.
"We're all in over our heads here, thanks to the deepening pool of technical debt previous decisions have left us with, and as a competent engineer who's quite capable of finding employment elsewhere on a project that may not have these problems, I'm far more afraid for the company and the customer than I am for myself."
Manager: And you're certainly not going to ruin it for me. Don't EVER use the phrase "political reasons" in a professional document.
"You don't have to pass this on. I'm just telling you the truth. If you'd like my help in getting the politics right, I'm happy to give it, but even an engineer understands organizational politics exist, and it's no use pretending they don't."
Tweet, tweet.
That 4k in java isn't 4k in freshman comp sci java. It's going to be leveraging widely used libraries that have millions of SLOC and a wide deployment base. Reading between the lines, I would be surprised if the tiger team didn't use the same tools as countless midsized projects that are maintained by teams of a few dozen at most. You can do a phenomenal amount in little code if you use the right frameworks and some decent xml configuration files.
In fact my coworkers and I have noticed an extremely frustrating trend in our side projects. We apply tools we learn at our day job (in part to understand them better), and our side project SLOC shrinks. A lot. Much of the 'interesting bits' disappears. Suddenly a 12k SLOC side project that took some serious effort to maintain is just a 4k SLOC side project, more functional, and easier to extend. It's a good thing -- don't get me wrong -- but it can be a real ego buster.
Portability isn't a big driver on server-side projects. It's nice, but only really comes into play when you deploy to new hardware or OS version. The larger the project, the longer the deployment cycle. (Think enterprise linux.) For something with a budget in hundreds of millions of dollars you'll have a deployment cycle that lasts at least 5 years.
For every complex problem there is an answer that is clear, simple, and wrong. -- H L Mencken
I have a theory about the missing senior managers with useful skills: My father talked about his teaching debate in high school in the mid '60 to late '70 in central California (then moving on to a community college), where he said that many of the most socially active "lets change the world for the better" students he ever taught went to Vietnam and never really came back, in one way or another. Those people would now be in their 60s, ten years older than I, and their influence would be reaching their peak. Those people are the leaders and mentors that I feel have been missing from the last third of the 20th century.
Sys Admins and Dev Gurus are not Project managers. But they get put in the position of being one constantly because upper management doesn't know the difference.
No, they get put into the position because they're inevitably the poor fuckers who have to try and pick up the pieces when the shit hits the fan - usually because they're the only ones who have the slightest inkling of how everything actually fits together.
Most sysadmins I have met have a fairly limited grasp on the actual business aspects of running a company. However, they have always had a better understanding of that, than any "manager" has of the technical aspects of the operation, that I have ever known.
Basically, IME, it's far more productive to try and teach a technical person management skills, than it is to teach a manager technical skills (or even concepts).
There seems to be plenty of blame to go around, but I think it's rather disingenuous for consultants to be rewriting code in Java instead of [original obscure language]. The comparison has no value, and I hope they didn't bill their client for the time. Unless they were brought in to do a Java rewrite, and that doesn't appear to be the case, they should have been spending their time working in [original obscure language].
Some of the things that stood out to me are things like "Even the current effort probably requires a year to get something into production, but the schedule says four months." They needed to refocus their energies on short term goals. I'm sure they have a list of things that need to be fixed or added to the project. When things started getting bogged down they should have looked at that list and said "what can we code/fix and test this week". Smaller milestones. It keeps the developers from getting beaten down, it gives management/users a deliverable, and it can change the tone of the project. If they have been working on this project for years with no success, anyone competent has left, or is looking for new employment.
And I have to wonder when the consultants say the staff developers don't have the expertise and are too numerous, are they taking into account business knowledge? I have worked with several consultants that were knowledgeable in the tools they were using, but couldn't translate their terminology into business terminology. Sometimes staff developers have to spend most of their time translating business requirements for consultants.
And finally, they were trying to change core methodology in the middle of a critical project? There is a lot to be said for Agile/RUP styles of development. They obviously weren't using that process before, and they obviously need some change. But it would be better to implement just a few of the more useful techniques (incremental releases, short term deliverables) rather than upsetting the whole development process at a critical time.
A fact of IT is that projects are rarely properly prepared. Missed requirements become feature creep. (Feature creep is blamed on users, missed requirements are IT's fault) If the benefits of Java (or some other language) over [original obscure language] is so high, maybe they should have considered changing languages at the start, but only if they are going to give proper training to the developers.
It is amazing what you can accomplish if you do not care who gets the credit. -- Harry Truman
I've seen screw-ups totaling hundreds of millions (that we knew about). Possibly billions if some forensic accountants could be sent in. I've worked on the fixes for a few of these. Some so simple that it took half a dozen people about 6 months to successfully build a replacement.
So, where does the money go? I mean $250 million USD on a f*ck-up? The perpetrators aren't driving Ferraris and Bentleys, so I doubt it was embezzled. In truth, project FUBAR was never actually shut down. It still exists as a server with some documents. This allows the company to depreciate the development costs and not have to take a write-down that Wall Street would notice. There's another reason the project, while brain-dead, has its corpse propped up at board of directors meetings:
This outfit is, among other things, a government contractor. They just lost a big one to a competitor, which they are appealing. Part of the reason for the loss is that gov't inspectors, having become aware of the true scope of woe, selected the better alternative. One (rare) example of the gov't actually looking out for our pocketbooks. But in the end, I suspect they (we, the taxpayers, that is) are destined to lose the appeal, since the project is not officially dead and can't be used as the basis of a negative performance report upon which the bid decision was based.
Have gnu, will travel.
I suspect this has more to do with dependencies than loss of features. It's easy to take an ancient project and lose lots and lots of code, usually because the old one does everything by itself and the new one brings in externally maintained libraries and frameworks. NIH doesn't even apply--if the project is old enough, Inventing it Here may have been the only option.
The rest of the difference can be made up by better coders and better coding techniques and technologies. Most rewrites produce better, smaller, faster code, this is just an extreme example.
It's rare that you're presented with a knob whose only two positions are Make History and Flee Your Glorious Destiny.
I meant to mention a slightly more hopeful thing that also sometimes happens in these situations. Senior management sometimes hires consultants to review a project like this, precisely because they suspect it really isn't going well. You talk to several folk at different levels, and they all tell you the same thing, which on the project described in the parent article would be something like, "it will fail unless several things radically change." Of course, they may represent several different visions about how to fix it, and what precisely is the problem causing the failure, though usually those overlap to some degree and are seldom mutually exclusive.
This can happen at all levels of the organization, the truth, more or less, might be an open secret. However, for reasons of internal politics and personalities, they need someone from the outside to be the one to tell them.
I once witnessed senior management bring in Peter Drucker, at great expense and inconvenience. Before he takes a gig like this, he basically tells the senior management that he gets unlimited access to anyone in the organization, or he's not interested. Then he starts talking to people, in private. He follows threads of interest. He doesn't spread gossip, and he doesn't make judgements, except those in the final report to his client, who is usually the CEO or Board of Directors. In conversation with people he spends almost all of his time listening. With a strategy like that, you can get to the truth of the matter, even though the truth is often very complex, from the standpoint of organizational politics, anyway.
Later, they replaced the CIO, partly on the basis of his report, which most likely didn't say anything like, "get a new CIO" anywhere. Rather it said something like, "this project has already failed, you just haven't realized it, yet," and "here are the indicators that it has failed," etc.
Watching that particular new CIO come in was fascinating, too. He was one of the best CIOs I've ever met. He spent about two months getting his bearings, using essentially the same techniques, only in a manner that seemed lower profile. He just visited with people at different levels of the organization, in the course of normal business, and without any apparent agenda. He didn't march in as a trumpeted turn around artist or anything like that. He was all business. Once he started making decisions, they were big decisions, and they were almost immediately recognizable as the right decisions -- or at least members of the class of possible right decisions for the given problem domain. It took a little while, but people who had been demoralized on this giant failing project (which had involved nearly the entire IT staff to greater and lesser extents) got back a sense of esprit de corps.
If you mod me down, I shall become more powerful than you could possibly imagine.
I have no problem believing that one.
,and I quote, "a 4 superdome cluster". They had just deployed it for a customer with the same load I was testing with. Superdome is the name for high-end HP unix boxes. You can safely assume that they weren't talking about the low-end.
5 years ago, working for a Big Company writing Billing software, I was summoned by management and got screamed at because the "module" I was coding, in Java, ran faster on my PC than the previous version of the app, written in C, on
And you're calling them out on it? You think they did the wrong thing by eliminating 136,000 lines of bloat and significantly improving the performance?
Absolutely they were wrong to do it. First, the code they wrote did NOT lead to any production code, either in Java or it's original code base. The project was killed. Second, they weren't expected to manage the code, they were hired to HELP complete the project. Third, they were hired to modify the existing code base, not create a new one.
The last thing a company needs out of a consultant is for them to waste time on something outside the bounds of the project and completely incompatible. If they couldn't do it in the scope of the project, they either should not have accepted the contract or at least offered a project plan to restart in Java. The customer had a specific reasons for doing things the way they did. It's not the consultant's place to berate them on that choice. Would it be acceptable to you to hire consultants to help with a C# project and have them rewrite a ton of it in Java and tell you it smells like roses? Java was the wrong approach for that stage of the project.
And you're making assumptions on the complexity of the code and whether those 4200 lines of Java actually did produce an acceptable alternative.
It is amazing what you can accomplish if you do not care who gets the credit. -- Harry Truman
By sheer chance (and blind luck), the pointer that overflowed was now writing into the padding. As the padding wasn't used for anything, it didn't result in data being overwritten or a segfault. The object files contain all of the static allocations, so it was obvious from the start that this was a static array that was causing the problems. Since no other variable got corrupted, it also had to be the last static array in the static space being allocated in an object file. (By "static", I merely mean not allocated off the heap, I do not mean variables prefixed as static and retained after exiting the function. In fact, it was also obvious it could not be a retained value as retained values are placed at the top of the static allocation because you'll otherwise risk overwriting them or fragmenting variables. Variables that exist at the time of calling a function but are still not considered dynamic grow down from the top of unused static space towards heap space.)
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
Not as simple as that. From experience with legacy code, it is hard to believe some good business rules were not lost and now have to be re-found and re-coded. I guess the consultants decide unilaterally what could go and stay. Let's hope they had previously committed the entire specification to memory.
I would be terrified if any consultants took it upon themselves to recode anything just for size and performance reasons.
The most valuable assets in a firm are mature reliable code and people who know how to maintain it. When those consultants are gone, BigFirm has been robbed of these assets.
Wow. Just... wow. Not wow, that all the stuff in the memo is happening, but wow, that this made it into a memo. This is what happened on every damn project in every damn company I've ever been in. Some projects do eventually get cancelled. Others just pass the finish line and limp on to deployment. Many of those reach a state of just being good enough, and then they cannot be maintained because they rot in a sense. The people who originally developed the thing are gone, and as knowledge gets transfered from one tech to another, it mutates until the product hopefully dies because nobody wants to use it.
I've spent nearly two decades in the software development industry, and after nearly 15 years of teeth-gnashing and 5 years of deep introspective thought, I've come up with two possible reasons. Up to you whether you like them or have your own ideas.
1. Most people are just not that intelligent. Perhaps you and I take to software very easily. Perhaps we think in C. We know exactly what the computer is going to do, and how to make it do that. And perhaps we are rare, which leads projects to be stuffed with the seemingly incompetent. It's not that they're incompetent -- it's that they are average!
2. Capitalism is a race to the bottom. Companies do not compete to provide a better product cheaper, because generally better and cheaper do not go together -- that's what science is, not business. Therefore, companies compete to provide a worse product cheaper. There is absolutely no incentive whatsoever to produce a good product, so why spent a lot of capital on excellent software developers?
I just can't come up with any more plausible reason for the crap churned out by the industry. I'd like to hear alternate ideas. I am very depressed that the average is so poor and that we live in a society that seeks mediocrity.
Yeah, from what I read the FBI project blew through $150 million or more, only to be told that there wasn't anything useful to be gained out the work that had been done.
..bruce..
Hadn't heard about McDonalds; I'll have to go digging for information on that one.
Bruce F. Webster (brucefwebster.com)
It's not legacy code if it has never been in production. Sometimes it *is* better to just throw the crap out and start over, even for parts of a project.
I'm a regular reader of TDWTF and I can't recall a single example of a gross exaggeration by Alex that has been called out by the contributor. There are often examples of misunderstandings and ambiguities that have to be corrected by the contributor, but i suspect that most of the exaggeration is done before Alex sees the story.
All I want is a secure system where it's easy to do anything I want. Is that too much to ask ~~ Randall Munroe