Slashdot Mirror
Anatomy of a Runaway Project
JCWDenton recommends a piece by Bruce Webster revealing some insights into a failed multi-million-dollar IT project. "The following document is the actual text — carefully redacted — of a memo I wrote some time back after performing an IT project review; names and identifying concepts have been changed to preserve confidentiality (and protect the guilty). The project in question was a major IT re-engineering effort for a mission-critical system; at the time I did this review, the project had been going on for several years and had cost millions of dollars; it would eventually be canceled and the work products abandoned. The memo itself provides an interesting glimpse into just how a major IT project can go so far off the tracks that nothing useful is ever delivered."
It's sort of interesting, in a vague way, but you can read much more dire and funny stories on (the highly recommended) the daily WTF. My favourites would have to be the hotel reservation system from hell, the story of VirtuDyne and the digital donkey and a case of the MUMPS.
I don't see a similarity, really.
Wine is actually an example of something extremely rare - a project that looked like it was doomed from the beginning, took millenia to get to the current state, but achieved usefulness anyway. Most of the time it works and when it doesn't, most of the time it's just common bugs, not incompleteness.
This is Slashdot. Common sense is futile. You will be modded down.
Damn. That's the exact phrase I've been looking for. I don't know how many times I've seen hard truths and unpleasant realities float up the organization and stop dead about 3 management levels below where someone could do something about it. Just like sonar, the people "listening" above the thermocline will never hear anything occurring below it.
Welcome to the Panopticon. Used to be a prison, now it's your home.
ObOwnExperience: One time, I had to do some maintenance work on a very large piece of badly-written and cruft-ridden code, ended up rewriting large tracts of it, reduced its source size by an order of magnitude and the binary size by three orders of magnitude. Also found some buffer overflow Heisenbugs which the previous maintenance guys had known about but bypassed by padding the object files. There's something... bothersome about corrupting a file in order to make a bug not be visible.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
"ISSUE: There isn't enough intellectual honesty within the FUBAR project. Managers reject or explain away bad news and real problems, looking instead for people who will tell them what they want to hear. "
This has easily been the #1 reason I have personally witnessed for project failure. I am in the process of witnessing it right now, even, with what seems like a relatively simple project. The suits and supervisors along the way are either not responding to requests for information, or change their request for features.
Bearded Dragon
On a project a few years ago (your typical Fortune 500 $LARGE_COMPANY here), our PM was forced to declare that a large release that had taken 6 months to get to "it's kinda working" was "complete" and shipped to UAT even though system testing was incomplete and all we did was give the end users a pile of steaming shit. But the director of the group got to "meet" his deadline, and therefore get his much-desire performance rating.
Of course fixing bugs once the app is in UAT is four times as difficult as in the integration environment, with the corresponding lag in defect correction time. So UAT went on and on and on... until it was supposed to be the final release date. Said users were hysterical and pissed off, and said director was out of his fucking mind trying to come up with inventive ways to ship said steaming pile of shit to production while blaming someone else for the smell of said pile.
His first executive action was to fire the PM and have him escorted out of the building (he was a contractor like me). His second action was to request that the Offshore Solutions team add four more developers to the already swelled-beyond-comprehension development team in India. You know, throwing bodies at the problem. The next thing was to go to his VP and claim that he had been misled by the PM, who by this time was checking out the classifieds at home and therefore unavailable for comment.
In the end, we all went through the usual death march and shipped the thing about three weeks late. The director got his rating (not his fault you understand) and the users had to deal with the remnants of the steaming pile of shit. I get paid either way so no skin off my ass.
Projects are late (or they fail) because the people who are supposed to be in charge of delivering them have no fucking clue as to how software is developed. Fix that problem, and you'll ship all the software you want on time and under budget.
I'm fortunate to be in a project right now where the guy in charge is a former developer who doesn't require a bonus to pay his mortgage, and all the two PMs do is move little bars on an MS Project file while mercifully leaving me and my team alone to actually write the software. We've released two major versions of the app so far the past two years and are on track to deliver the third version sometime this October. On time and under budget. The secret? Iterative development (SCRUM-like) with heavy user involvement in feature sprints. No waterfall bullshit for me anymore, thank $DEITY.
Web2.0: I love when people Flickr my cuil and digg my boingboing until my google is reddit and I start to yahoo
We've all gotten burned on projects that got out of hand, but I often wonder why it happens, over and over. Hubris?
I've seen projects where the requirements document was 1000 pages and growing exponentially. For an email server. I remember one project where it didn't matter if code even compiled: we had to ship what we had, because we couldn't delay any further. I made the mistake of expressing my views to the wrong people on that one, and was told in no uncertain terms to shut up if I wanted to continue working there.
I've seen more than one project fall flat on its face because the original requirements were wrong, like trying to develop PC software for an industry that was 100% Mac.
...laura
I have a theory about the missing senior managers with useful skills: My father talked about his teaching debate in high school in the mid '60 to late '70 in central California (then moving on to a community college), where he said that many of the most socially active "lets change the world for the better" students he ever taught went to Vietnam and never really came back, in one way or another. Those people would now be in their 60s, ten years older than I, and their influence would be reaching their peak. Those people are the leaders and mentors that I feel have been missing from the last third of the 20th century.
I've seen screw-ups totaling hundreds of millions (that we knew about). Possibly billions if some forensic accountants could be sent in. I've worked on the fixes for a few of these. Some so simple that it took half a dozen people about 6 months to successfully build a replacement.
So, where does the money go? I mean $250 million USD on a f*ck-up? The perpetrators aren't driving Ferraris and Bentleys, so I doubt it was embezzled. In truth, project FUBAR was never actually shut down. It still exists as a server with some documents. This allows the company to depreciate the development costs and not have to take a write-down that Wall Street would notice. There's another reason the project, while brain-dead, has its corpse propped up at board of directors meetings:
This outfit is, among other things, a government contractor. They just lost a big one to a competitor, which they are appealing. Part of the reason for the loss is that gov't inspectors, having become aware of the true scope of woe, selected the better alternative. One (rare) example of the gov't actually looking out for our pocketbooks. But in the end, I suspect they (we, the taxpayers, that is) are destined to lose the appeal, since the project is not officially dead and can't be used as the basis of a negative performance report upon which the bid decision was based.
Have gnu, will travel.
I meant to mention a slightly more hopeful thing that also sometimes happens in these situations. Senior management sometimes hires consultants to review a project like this, precisely because they suspect it really isn't going well. You talk to several folk at different levels, and they all tell you the same thing, which on the project described in the parent article would be something like, "it will fail unless several things radically change." Of course, they may represent several different visions about how to fix it, and what precisely is the problem causing the failure, though usually those overlap to some degree and are seldom mutually exclusive.
This can happen at all levels of the organization, the truth, more or less, might be an open secret. However, for reasons of internal politics and personalities, they need someone from the outside to be the one to tell them.
I once witnessed senior management bring in Peter Drucker, at great expense and inconvenience. Before he takes a gig like this, he basically tells the senior management that he gets unlimited access to anyone in the organization, or he's not interested. Then he starts talking to people, in private. He follows threads of interest. He doesn't spread gossip, and he doesn't make judgements, except those in the final report to his client, who is usually the CEO or Board of Directors. In conversation with people he spends almost all of his time listening. With a strategy like that, you can get to the truth of the matter, even though the truth is often very complex, from the standpoint of organizational politics, anyway.
Later, they replaced the CIO, partly on the basis of his report, which most likely didn't say anything like, "get a new CIO" anywhere. Rather it said something like, "this project has already failed, you just haven't realized it, yet," and "here are the indicators that it has failed," etc.
Watching that particular new CIO come in was fascinating, too. He was one of the best CIOs I've ever met. He spent about two months getting his bearings, using essentially the same techniques, only in a manner that seemed lower profile. He just visited with people at different levels of the organization, in the course of normal business, and without any apparent agenda. He didn't march in as a trumpeted turn around artist or anything like that. He was all business. Once he started making decisions, they were big decisions, and they were almost immediately recognizable as the right decisions -- or at least members of the class of possible right decisions for the given problem domain. It took a little while, but people who had been demoralized on this giant failing project (which had involved nearly the entire IT staff to greater and lesser extents) got back a sense of esprit de corps.
If you mod me down, I shall become more powerful than you could possibly imagine.
It's not legacy code if it has never been in production. Sometimes it *is* better to just throw the crap out and start over, even for parts of a project.