Slashdot Mirror
Anatomy of a Runaway Project
JCWDenton recommends a piece by Bruce Webster revealing some insights into a failed multi-million-dollar IT project. "The following document is the actual text — carefully redacted — of a memo I wrote some time back after performing an IT project review; names and identifying concepts have been changed to preserve confidentiality (and protect the guilty). The project in question was a major IT re-engineering effort for a mission-critical system; at the time I did this review, the project had been going on for several years and had cost millions of dollars; it would eventually be canceled and the work products abandoned. The memo itself provides an interesting glimpse into just how a major IT project can go so far off the tracks that nothing useful is ever delivered."
Not even 3 comments before the article was \.'ed to death.
Other than the fact they are delivering something...
"Let's not bicker about who killed who." Monty Python
The site is acting like it is soon to be slashdotted...
Anatomy of a runaway IT project
By bfwebster on Jun 16, 2008 in Main, Management, Project Failure
[Welcome to reddit and FARK visitors!]
The following document is the actual text -- carefully redacted -- of a memo I wrote some time back [i.e., several years ago] after performing an IT project review; names and identifying concepts have been changed to preserve confidentiality (and protect the guilty). The project in question was a major IT re-engineering effort for a mission-critical system; at the time I did this review, the project had been going on for several years and had cost millions of dollars; it would eventually be canceled and the work products abandoned. The memo itself provides an interesting glimpse into just how a major IT project can go so far off the tracks that nothing useful is ever delivered.
Note that the "ABC" consultants were a small part of the overall project team and had been brought in relatively late by "BigFirm" in an attempt to get the "FUBAR" project into production; they neither initiated nor managed the project. [NOTE for those of you who have written or done Google searches: "Bob Winsom", like all the other names in the memo as transcribed below, is a pseudonym.]
CONFIDENTIAL MEMORANDUM -- EYES ONLY
Over the past two weeks, I've conducted confidential off-site group interviews with all of the ABC consultants working on the FUBAR project. I did this at [ABC manager's] request, after a few of these consultants spoke privately about FUBAR with him. The feedback was consistent and raises serious doubts about whether the FUBAR project, as currently pursued, can ever yield a successful production deployment.
This report groups those comments into several broad areas. It is relatively unfiltered and extremely direct (no withholding). It represents the private comments of ABC consultants who have little to gain and possibly much to lose by being so blunt. These are not the whinings of purists picking nits. These are the grounded assessments of top-notch IT professionals who have among them a century or two of experience bringing projects to completion -- particularly those involving [specific IT] technology -- and who are down in the FUBAR trenches every day. QUALITY OF WORK AND EFFORT
ISSUE: Several consultants said -- and the rest pretty much agreed -- that far too many of the deliverables, artifacts, and activities (e.g., algorithms, source code, system configuration, design/architecture documents, testing, defect tracking, scheduling etc.) are substantially below any acceptable professional standards and represent a profound threat to FUBAR ever going into production.
EXAMPLES: The code base is very fragile. A lot of it is bad old code that BigFirm didn't have time to rewrite two years ago, but now is five times its original size and even worse. One consultant said he took a code listing, picked pages at random, and found problems on every page he selected. There is pervasive hard coding of what should be adjustable parameters or at least meaningfully named constants (e.g., # of [key items] hard-coded throughout with the literal value '3, a constant named 'ninety_eight'). Builds take all night. App releases don't run acceptably, if at all, in a production environment. Developers check in files that won't even compile.
RISKS: The FUBAR project keeps being touted as a world-class development team, but it is not producing world-class, or even minimally-professional, results. This already shows up in the project delays and quality issues of the releases to date. What the team is producing will not only be very difficult to support and modify, it will in all likelihood be unusable, resulting in a complete failure of the FUBAR project. PROJECT PLANNING AND EXECUTION
ISSUE: Project planning and execution is all to often poor or missing completely. Milestone dates, usually unrealistic if not impossible, are based on p
No Nyarlathotep, No Chaos
Know Nyarlathotep, Know Chaos
How about Anatomy of a slashdotted server?
Look where all this talking got us, baby.
This link hasn't been slashdoted: http://209.85.165.104/search?q=cache:vTIf3RxPVXAJ:brucefwebster.com/2008/06/16/anatomy-of-a-runaway-it-project/+anatomy+of+a+runaway+project&hl=en&client=firefox-a&gl=us&strip=1
I dunno, for it to be ironic Wine would have to have shared some of those characteristics, but it really doesn't.
In particular, the key problem with FUBAR project appeared to be Mr Bob Winsom, whoever he is, who was clearly not technical or competent but believed he was. Wine is led by Alexandre Julliard, who is every bit as competent and skilled as Linus Torvalds himself, if not moreso, the primary difference being that Linus quite a loud person and AJ is not.
Wine has taken a long time to reach 1.0 (a rather arbitrary line in the sand) because Windows is a huge codebase, which is very difficult to match exactly to the expectations of the apps running on it. At its peak Windows had over 5000 engineers working full time on it, something Wine has never had.
I view these problems as a direct result in regards to a lack of IT project managers. Even being a seasoned IT professional I wouldn't even begin to imagine the difficulties in managing a large IT project beginning to end. In my experience I have noticed in run-away projects or in slow moving projects, the problems usually are attributed to a change of project managers. I would urge any company with an IT project to do whatever they can to maintain their PMs with any incentives possible. Because the fallout is financially distressing to ANY company.
CS: It is all sink or swim...oh and did I mention there are sharks in that water?
Was he singled out for a reason ?
Come on now, COBOL isn't that bad. :P. But seriously Java isn't the language you would use for high performance but rather high portability. That says a lot about how bad the original code was.
Well, there's spam egg sausage and spam, that's not got much spam in it.
or at least a hint, so we can understand better the reality we live in. It could as well be SCO or some mediocre company - in that case, it would be meaningless. But what if it was Yahoo! or Microsoft?
From TFA:
The FUBAR project keeps being touted as a world-class development team, but it is not producing world-class, or even minimally-professional, results. This already shows up in the project delays and quality issues of the releases to date. What the team is producing will not only be very difficult to support and modify, it will in all likelihood be unusable, resulting in a complete failure of the FUBAR project.
Sounds like Vista to me...
That "FUBAR" project is Duke Nuke Em Forever.
I am Bennett Haselton! I am Bennett Haselton!
Since its already \.'d Note that the âoeABCâ consultants were a small part of the overall project team and had been brought in relatively late by âoeBigFirmâ in an attempt to get the âoeFUBARâ project into production; they neither initiated nor managed the project. [NOTE for those of you who have written or done Google searches: "Bob Winsom", like all the other names in the memo as transcribed below, is a pseudonym.]
CONFIDENTIAL MEMORANDUM â" EYES ONLY
Over the past two weeks, Iâ(TM)ve conducted confidential off-site group interviews with all of the ABC consultants working on the FUBAR project. I did this at [ABC manager's] request, after a few of these consultants spoke privately about FUBAR with him. The feedback was consistent and raises serious doubts about whether the FUBAR project, as currently pursued, can ever yield a successful production deployment.
This report groups those comments into several broad areas. It is relatively unfiltered and extremely direct (no withholding). It represents the private comments of ABC consultants who have little to gain and possibly much to lose by being so blunt. These are not the whinings of purists picking nits. These are the grounded assessments of top-notch IT professionals who have among them a century or two of experience bringing projects to completion â" particularly those involving [specific IT] technology â" and who are down in the FUBAR trenches every day.
QUALITY OF WORK AND EFFORT
ISSUE: Several consultants said â" and the rest pretty much agreed â" that far too many of the deliverables, artifacts, and activities (e.g., algorithms, source code, system configuration, design/architecture documents, testing, defect tracking, scheduling etc.) are substantially below any acceptable professional standards and represent a profound threat to FUBAR ever going into production.
EXAMPLES: The code base is very fragile. A lot of it is bad old code that BigFirm didnâ(TM)t have time to rewrite two years ago, but now is five times its original size and even worse. One consultant said he took a code listing, picked pages at random, and found problems on every page he selected. There is pervasive hard coding of what should be adjustable parameters or at least meaningfully named constants (e.g., # of [key items] hard-coded throughout with the literal value â3â, a constant named âninety_eightâ(TM)). Builds take all night. App releases donâ(TM)t run acceptably, if at all, in a production environment. Developers check in files that wonâ(TM)t even compile.
RISKS: The FUBAR project keeps being touted as a world-class development team, but it is not producing world-class, or even minimally-professional, results. This already shows up in the project delays and quality issues of the releases to date. What the team is producing will not only be very difficult to support and modify, it will in all likelihood be unusable, resulting in a complete failure of the FUBAR project.
PROJECT PLANNING AND EXECUTION
ISSUE: Project planning and execution is all to often poor or missing completely. Milestone dates, usually unrealistic if not impossible, are based on political considerations or wishful thinking, not bottom-up grounding. Necessary and useful activities are delayed or canceled with the justification âoeWe have no time for thatâ, but the project phase ends up taking as long or longer than if the activities had been carried out. Dates are set, but nobody scrambles until the last minute. Risks are not actively tracked or managed.
EXAMPLES: Count how many times FUBAR ever produced a production-quality deliverable on anything close to a scheduled date. Even the current effort probably requires a year to get something into production, but the schedule says four months. Managers create work tasks, but then never track progress or completion. One ABC consultant created a risks
The runaway project was the web server this story was hosted on....
Two consultants rewrote the 140,000 lines of [original obscure language] into 4200 lines of Java. The Java version runs as fast on a laptop PC as the original version runs on a high-powered UNIX server
Okay, ummmm, I'm gonna need some more detail on this statement.
I first read this as "Anatomy of Runway Project" and thought of Heidi Klum.
I am *so* disappointed with the actual article...
It must have been something you assimilated. . . .
IT budget and spending has a negative correlation with efficiency of operation of whole enterprise.
Sounds like Project ASPIRE in Florida. 70+ million pissed away to replace the aging mainframe state accounting system. That project had everything - a corrupt politically connected contractor, inept management, you name it.
The thing is, 5,000 engineers horsing around isn't the same thing as a 5,000 horsepower engine.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
I don't see a similarity, really.
Wine is actually an example of something extremely rare - a project that looked like it was doomed from the beginning, took millenia to get to the current state, but achieved usefulness anyway. Most of the time it works and when it doesn't, most of the time it's just common bugs, not incompleteness.
This is Slashdot. Common sense is futile. You will be modded down.
Then there's that person who's "kind of a big deal" and thinks the project is "fierce". That would be the senior administrator.
Oh wait. This isn't Project Runway"?
Rewriting 14,000 lines of the project in [obscure language] as ~2000 lines of Java, and that the product ran on high end Unix servers.
I have a dedicated server and have had slashdotted postings before that haven't brought the server down. I've e-mail the support team to see what's going on. ..bruce..
Bruce F. Webster (brucefwebster.com)
I'm sure that there are other similar companies out there, but much of the language and all of the circumstances seem very familiar. Just curious, but how many other companies use the term "deliverables"? IBM, after purchasing Rational Software, decided that it was a good idea to move all projects to this process. About 2003, there was a huge stir within the company to document everything into a "process" and wasted months (nay, years?) in fluff process documentation that yielded no benefit. It is very interesting that their stock is doing so well at the moment. Lots of folks are jumping ship like mad right now . . .
StyleChief
Strange women lying in ponds distributing swords is no basis for a system of government! -M. Python
no reason for the fud-meisters to get their knickers in a twist. enough hysterics, lets just let the highly qualified smart people do their jobs.
This report is little more than a walk through of the textbook examples that cause project failures, and what the possible consequences are. Interesting, but nothing new to those already in the industry. A lot of these symptoms are present in every project (not just the lost-cause ones). Most can attest to this.
With some more detail this firm would make a great textbook example for future students. What was the end result of this project, or is it still active?
For he today that sheds his blood with me shall be my brother.
In any case, the project to which Mr Webster refers is clearly Microsoft Windows Vista.
Rich And Stupid is not so bad as Working For Rich And Stupid.
I've seen this so many times with the big consulting firms. They low ball the bids. Then they send in kids who were just handed their degrees and a manual about some technology and told to implement the technology at a client's site (basically it's the only people they can afford with the bid). It goes downhill because their lack of experience and lack of project management. Later the big consulting firm brings in a subcontractor to fix the issues (mainly because the client refuses to pay anymore and may have milestones/clauses that allow them not to pay). The subcontractor usually is smaller, has more expertise, but costs much more. But they are given a huge and seemingly impossible task. Sometimes they can rescue the project. Sometimes they can't.
Well, there's spam egg sausage and spam, that's not got much spam in it.
approach to project development. "If we hire twice as many programmers, we'll finish it in half the time!"
I ran into this career driven mid-level manager problem-solving approach regularly in the 90's before many of them vaporized (remember DEC?) Time has not changed human nature or incompetent managers.
The PM's of these projects tended to be big on contrived dog-and-pony shows too as I recall.
But seriously Java isn't the language you would use for high performance but rather high portability
That old myth? That hasn't been true for many years now, for server side code anyway (which this was describing). Modern JIT compilers make java as fast, and sometimes faster (since you are optimizing code as it runs and not statically beforehand).
But no language will help you if you lack discipline or the ability to code (both of which seemed lacking in this case).
"There is more worth loving than we have strength to love." - Brian Jay Stanley
A cursory glance at a semi-reliable source of information hints at which company ABC just might be (also apparently on the same slashdotted server as brucefwebster.com -- confirmed via DNS).
Now who would like to identify BigFirm?
FUBAR=ASPIRE
ABC=Gartner Group
BigFirm=BearingPoint
I wonder if their server is the failed project...
Circumcision is child abuse.
It could well be running up to an implosion (revenues won't necessarily reflect erosion of long term business), but the stock is up on real revenue and earnings growth:
http://finance.yahoo.com/q/is?s=IBM&annual
Nerd rage is the funniest rage.
Just curious, but how many other companies use the term "deliverables"?
All of them. Phrases like that are highly viral and travel through the world management population in under a month.
The article was so generic, it could have described projects I've seen from companies with under a hundred people to a cast of thousands.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
From large companies to small, it's the single biggest problem they all have. Decisions makers don't want to hear the truth, no matter how loudly they protest otherwise. Anyone with intellectual honesty that doesn't have a previously won huge level of trust from a decision-maker is almost invariably thought to be lying. They all want to have their cake and eat it too, and they will throw money at anyone that tells them they can. Even after getting burned by the consequences of their decisions, less than half (in my experience) bother to try and learn from the failure. Most of them blame the honest person (if they did nothing and as a result failure happened) or latch on to the next person willing and able to lie even more convincingly than the last guy.
It is unfortunately common to knee-jerk a development problem by adding more management.
A project manager who doesn't actually have the skills it takes to make a project successful will be as bad or worse than no project manager at all. Hiring/retaining more of them will just multiply the problem.
It is very difficult to interview for and find good project managers. The talent pool is just teeming with people who are not skilled developers, and would to love to have a job that is, essentially, just telling other developers to do their jobs. There is tremendous incentive for people who are not competent to be a project manager (or much of anything else, for that matter) to fight tooth and nail for PM jobs. When you hire such a person, your project usually fails, or if it does succeed it is despite, and not because of, the best efforts of your project manager.
Another problem: the best project manager in the world won't get you results if you disempower him or her. I have seen it happen often that the executives see a project slipping and shift into micromanagement mode. At that point, the project manager just becomes a mouthpiece, and the company has robbed itself of the value of their paid talent. If you can't trust your project manager to tell you when a goal cannot be achieved, or when more time must be allocated to some task that doesn't have obvious functional benefits, or that a deadline must be extended, then you have either hired a lemon or you are involving yourself too much in his job. In either case, your project will suffer because of it.
Ok, I will stop ranting now. The bottom line is...more management doesn't solve problems. The right amount of *competent* and *properly empowered* management does.
I'm a bit stunned myself. The article had been picked up by reddit and FARK prior to Slashdot, but even so, I've had another post on my blog previously linked to by Slashdot without server problems. I've already contacted my hosting company to find out what's going on.
Bruce F. Webster (brucefwebster.com)
Agile software development is exactly what should be the solution to such mistake.
Here in Cleveland they have a new runway project at Hopkins that should be kicking off soon. 6L-24R, 7000 ft long, 150 ft wide. Quite an undertaking!
For all the crap output may equal that of 250 score horses.
Get thee glass eyes, and, like a scurvy politician, seem to see things thou dost not.--King Lear
The example is far to familiar, I have worked with lots of successfull projects and also to "dead" multimilion multiyear projects, there are some basic differences:
- the dead project owners/managers gain more by the hours of coding, than by the results. Example: an internal project who's manager wants to prolong it for years to get an easy paycheck and have an architect aura. Or a consulting project for a high paying customer (a bank or government in best case scenario), obtained as usually on relations.
- the succesfull project owners/managers gain by results (a project negociated as a total sum, not as development time, or a product/service that the software company sells)
The rest is history (the two cases may occur at the same company, on different projects).
Unfortunately for the author of this memo...it seems to lack credibility.
Almost everything stated is based on opinion. It reeks of "amateur", and would be ripped apart by just about any manager it was given to. Here I will show paraphrased examples of what was written by a managers reaction to reading it:
Memo: It has grown too complex to ever go to production
Manager: Please outline the facets of this project that can be eliminated.
Memo: Some coder on my team found a problem with every page of code he sampled
Manager: What makes this coder more qualified than the coder who wrote it?
Memo: The code base is "very fragile"
Manager: What the fuck does that mean?
Memo: My guys took 140,000 lines of old crappy code and replaced it with 4200 lines of Java
Manager: This guy is one of those "I can do it better using the new language i learned in college" kids
Memo: Many previous project managers have left or not been given the power to architect it properly
Manager: This guy is obviously in over his head and fears his job.
Memo: This project has grown too big, probably due to policial reasons of some guy wanting a big important project
Manager: And you're certainly not going to ruin it for me. Don't EVER use the phrase "political reasons" in a professional document.
Check out the story here.
A nice little 3.5 year IT boondoggle that cost a cool $70 million and cost one board member of 19 years his job. It all just came to light last month. It made some pretty big headlines around these parts as well.
Every company I've worked at over the last 15 years has used this term -- from tech companies, to marketing agencies, to international nonprofits. It's universal.
$nice = $webHosting + $domainNames + $sslCerts
Like I always say: Winsom, lose some.
stuff |
We're killing the Internet!
"Three eyes are better than one" -- Lieutenant Columbo
That said, the article seems to be hosted on his own site where, conceivably, he has posted his resume. Couldnt someone simply research the companies listed against the article me mentions here and arrive at a pretty good idea just which company is being discussed in gory detail here?
Sig Follows: "Suppose you were an idiot. And suppose you were a member of Congress. But I repeat myself." -- Mark Twain
Only the government waste's money. Corporations don't. When corporations spend money on projects, even when they fail, they are *INVESTING* in their future. There is a huge difference here, but slashdot is full of idiot statists who wouldn't understand.
We've all gotten burned on projects that got out of hand, but I often wonder why it happens, over and over. Hubris?
I've seen projects where the requirements document was 1000 pages and growing exponentially. For an email server. I remember one project where it didn't matter if code even compiled: we had to ship what we had, because we couldn't delay any further. I made the mistake of expressing my views to the wrong people on that one, and was told in no uncertain terms to shut up if I wanted to continue working there.
I've seen more than one project fall flat on its face because the original requirements were wrong, like trying to develop PC software for an industry that was 100% Mac.
...laura
Thanks. Now I won't be able to sleep until I know what "[original obscure language]" was used.
You are where you are at the time you are there.
The difference (I'll bet) is that the prior link included enough of the original article for the vast majority of slashdotters to just read that and then dive into the discussion.
I'll freely admit that I only RTFA on about 5% of Slashdot stories. This story was one of those where I was on the way to follow the link, because it sounded very interesting not just for the subject matter, but for the actual content.
Custom, hands-free Linux installs. Instalinux
. It reeks of "amateur", and would be ripped apart by just about any manager it was given to.
I have a high degree of confidence that many managers could probably think they were ripping it apart, but my guess is your average slashdot poster (let alone your average decent engineer) could probably respond competently to each charge, were said manager competent enough to have the responses you gave be real commentary rather than contrary rhetoric.
Manager: Please outline the facets of this project that can be eliminated.
"This is part of the problem. The project has been so poorly organized and tracked that no one has a current of outline it. It's possible that we *can't* outline it."
Manager: What makes this coder more qualified than the coder who wrote it?
"As you'll see I mentioned, the developer reviewing the code was aware of widely-known good practices in development -- such as the use of well-named constants, rather than 'magic numbers.' The coder writing the code was either unaware of these or unwilling to apply them, which quite likely means he's less qualified."
Manager: What the fuck does [fragile] mean?
"It means that adding new features without breaking existing ones is likely to be difficult. The poor organization makes it easy to accidentally step on something important to who-knows-how-many other places in the code. It also means the application doesn't have broad ability to handle the set of all possible inputs robustly -- there are enough cases that aren't anticipated in the code (or may not be anticipated -- it's hard to tell with the poor oranization) that it will likely crash regularly."
Manager: This guy is one of those "I can do it better using the new language i learned in college" kids
*rolls eyes* -- the idea that a manager who'd say this is common is a complete flight of fancy. Where the hell are you going to find a manager in corporate America who is hostile to Java and sees it as something "new" that's the province of green college kids?
Manager: This guy is obviously in over his head and fears his job.
"We're all in over our heads here, thanks to the deepening pool of technical debt previous decisions have left us with, and as a competent engineer who's quite capable of finding employment elsewhere on a project that may not have these problems, I'm far more afraid for the company and the customer than I am for myself."
Manager: And you're certainly not going to ruin it for me. Don't EVER use the phrase "political reasons" in a professional document.
"You don't have to pass this on. I'm just telling you the truth. If you'd like my help in getting the politics right, I'm happy to give it, but even an engineer understands organizational politics exist, and it's no use pretending they don't."
Tweet, tweet.
That's the first time in 1,640 posts you've said "Winsom, lose some", you big fibber.
Get thee glass eyes, and, like a scurvy politician, seem to see things thou dost not.--King Lear
I think it all depends on how Mr Webster defines "success" (and "failure.")
No doubt Winsom is an idiot, but getting rid of him would be about as effective as capital punishment is in eliminating murder (i.e., not very). The problem appears to be systemic and pervasive, like poverty or police brutality.
How can this problem be solved? I have few ideas and little hope. Gall's book The Systems Bible presents some interesting insights.
The one ray of hope currently is FLOSS, whose projects are often free of this particular sort of nonsense. The big problem, of course, is that there seems to be no good, general way to compensate good people for working on these projects...
"Not an actor, but he plays one on TV."
Of course, a lot of people don't really want to be effective in the real world, not as much as they want other things. They want to feel good. They want everybody to like them. They want a quiet life. They want to keep collecting their paycheck. So they stick their heads in the sand and hum the national anthem.
Not that those are bad things to want. But you can't get them just by always picking them in the short term. Easy years require hard moments.
The UK NHS and ID card systems promise to completely blow tens of billions of GBP. But hey, it's only tax.... That's like, free, right.
One of the reasons i'm not too worried about ID cards.
Deleted
I read this with interest as I have been involved in large scale IT development projects for various corporations for the better part of 15 years. This memo makes it appear as if the problems in the project were execution related: bad management, poor quality control, bad architecture, performance problems, etc.
In my experience, it is actually not that common for an experienced team to fail largely on execution problems. Rather, as I like to say (call it Renn's Law if you'd like): "Most failed corporate software projects failed before the kickoff meeting". Usually the signs of failure were there all along, before the project even officially got started.
Here are some of the key things I've seen lead to problems, most of which are not directly related to the core development (design, build, and test) of the project:
- Lack of an identified executive sponsor
- Failure to identify a limited subset of people who are empowered and responsible for articulating the business requirements of the system
- Lack of clarity as to the actual goals to be achieved or the underlying problem to be solved.
- No shared vision of what a successful outcome would look like among the various stakeholders
- Project positioned as an IT-centric solution to a business-centric problem without a corresponding business strategy, process, and change management plan in place.
- Insufficient resources (time, money, people) allocated to the project
- Lack of qualified staff in key roles (data architect, functional lead, etc)
- Poor governance and scope control
That 4k in java isn't 4k in freshman comp sci java. It's going to be leveraging widely used libraries that have millions of SLOC and a wide deployment base. Reading between the lines, I would be surprised if the tiger team didn't use the same tools as countless midsized projects that are maintained by teams of a few dozen at most. You can do a phenomenal amount in little code if you use the right frameworks and some decent xml configuration files.
In fact my coworkers and I have noticed an extremely frustrating trend in our side projects. We apply tools we learn at our day job (in part to understand them better), and our side project SLOC shrinks. A lot. Much of the 'interesting bits' disappears. Suddenly a 12k SLOC side project that took some serious effort to maintain is just a 4k SLOC side project, more functional, and easier to extend. It's a good thing -- don't get me wrong -- but it can be a real ego buster.
Portability isn't a big driver on server-side projects. It's nice, but only really comes into play when you deploy to new hardware or OS version. The larger the project, the longer the deployment cycle. (Think enterprise linux.) For something with a budget in hundreds of millions of dollars you'll have a deployment cycle that lasts at least 5 years.
For every complex problem there is an answer that is clear, simple, and wrong. -- H L Mencken
http://en.wikipedia.org/wiki/Software_prototyping/ has long been known to produce something that looks done while running more quickly and needing less LOC than the real product will. One of the classic objections to creating a prototype as part of Rapid Application Development has been that management and/or the users would see the prototype and declare "looks great! ship it now!".
Pareto http://en.wikipedia.org/wiki/Pareto_principle/ applies as much to LOC as it does to optimizing performance.
You must be new here Tex, so I'll go easy. Just about every company with a project uses this term. I've mostly worked in the ERP world (for a looong time) and don't remember any but the smallest project where this term wasn't used.
This is not the first such discussion. Fred Brooks' book The Mythical Man Month contains some nice insights on IBM's OS 360 project and other such runaway failures.
It is surprising how many major IT groups keep relearning some of the same lessons, if they learn at all.
If my server ever recovers, go look at this earlier post defining and discussing the "thermocline of truth". ..bruce..
Bruce F. Webster (brucefwebster.com)
That could well be the case. My server appears to be topping out around 1550 simultaneous users (that's the highest I've seen via my SiteMeter info); I honestly don't know whether the previous article got up that high.The difference (I'll bet) is that the prior link included enough of the original article for the vast majority of slashdotters to just read that and then dive into the discussion.
Bruce F. Webster (brucefwebster.com)
*Always* put salt in your internet!
It is pitch black. You are likely to be eaten by a grue.
I'm a young developer(1 year out of college) who has been tasked with completing a rather large system.
The system was started 4 years ago, and is written in this thing called indusoft: http://www.indusoft.com/indusoft_webstudio.asp
The day I started here the project was already 1-2 years over due. Even today parts of it are still about 1 year overdue.
The kicker is I explained to them when I started that Indusoft was bad news. It's expensive $6,000 per license and a terrible deelopment tool. It has Object Oriented programming, only has one dimensional arrays, is buggy as all hell and has no debugger except trace statments(basically print statements).
Now, when I started I explained they had it all wrong. They of course didn;t listen insisting Indusoft was the best in teh market. Now, we are still not done. Still not hitting our mark, and have a maintenace nightmare.
We have something on the order of 250,000 lines of VBScript. It had passed through the hands of a half dozen developers. We have no spec, no test cases, no schedule, no milestones and no goal except "finish it".
What a nightmare...
Yes, and unfortunately, the typical bureaucracy enforces this thermocline, brutally. People get fired, not for wasting years and tens of millions of dollars through incompetence, nor for hiding the truth, nor for outright lying about the true status of a failing project, but rather more often, for attempting to fix problems by trying to go around a roadblock that is preventing the truth from going up the corporate ladder. When they are not outright fired, they are certainly blacklisted, and get to watch, as their own career stagnates, and the careers of those erecting the roadblocks advances. As a consultant, I've had the opportunity to see this happen to highly competent, dedicated, organizationally loyal people who should have been given million dollar bonuses, and promoted to Director, or CIO. It's quite sad, but I could never recommend to anyone to be the voice of reason on a project thats failing like this, unless they are prepared to lose their job.
If you mod me down, I shall become more powerful than you could possibly imagine.
Fire "Bob Winsom."
Curse you for not porting it to the Amiga! That was the one game on the ST that made me (almost) think about buying an Atari.
Learning HOW to think is more important than learning WHAT to think.
The article points out that project management didn't do grass roots valuation or [pause] management. I would bet that the PM for this project has a PMP cert. As a former tech lead/architect for a death march type project, I can honestly say sometimes people in roles such as mine do have the expertise to advise management of risk, complexity, and time/cost. The biggest obstacle is the mind-set and experience of the PM and their bosses. For example, the last project, I mentioned certain things were high risk and/or would take significant time due to complexity and "unknown dangers." The question put to me was "Can we solve this by throwing more people at the problem without changing the time-line [that was picked out of the air]?" My response was, "Its the mythical man month." The blank stare I had in response forced me to explain the concept of overhead logistics, ramp up time, and cost/time trade-off features and risk. [The unfortunate side effect of me educating the PM about her job made me a threat and obstacle to be removed.]
You can have a kick azz senior team and still fail.
/\/\icro/\/\uncher
I've seen screw-ups totaling hundreds of millions (that we knew about). Possibly billions if some forensic accountants could be sent in. I've worked on the fixes for a few of these. Some so simple that it took half a dozen people about 6 months to successfully build a replacement.
So, where does the money go? I mean $250 million USD on a f*ck-up? The perpetrators aren't driving Ferraris and Bentleys, so I doubt it was embezzled. In truth, project FUBAR was never actually shut down. It still exists as a server with some documents. This allows the company to depreciate the development costs and not have to take a write-down that Wall Street would notice. There's another reason the project, while brain-dead, has its corpse propped up at board of directors meetings:
This outfit is, among other things, a government contractor. They just lost a big one to a competitor, which they are appealing. Part of the reason for the loss is that gov't inspectors, having become aware of the true scope of woe, selected the better alternative. One (rare) example of the gov't actually looking out for our pocketbooks. But in the end, I suspect they (we, the taxpayers, that is) are destined to lose the appeal, since the project is not officially dead and can't be used as the basis of a negative performance report upon which the bid decision was based.
Have gnu, will travel.
Wine's the sort of huge, complicated, moving target that one might expect to spend millions of dollars and many thousands of engineer-hours on and end up with nothing to show.
Yet somehow, with just a small team and without the millions of dollars poured into it, they somehow managed to come out with an actual working product.
And here I thought I was going to be reading an article on Heidi Klum...
I know of a company that after spending about $1.5Mil or so re-engineering their current environment, and had it all ready to deplay - they pulled the plug on it and decided to go with an SAP solution. Not that SAP had ever done a "best practices" in this industry (that I know of). So, they will end up letting go a bunch of their IT people and "save money" by hiring a couple of dozen SAP people.
Assuming they can stay on schedule and on budget (is that possible with SAP?).
Makes me wonder though why it is that these kind of things only seem to occur in the software industry? Ever heard of a shipping yard abandoning the building of ship? (Guess I should expect any anecdotes coming this way now).
Is it the fact almost every major s/w project is pretty unique? Or the industry is willing to put fresh out of college engineers straight onto the project?
Perhaps when it comes down to it a ship or anything physical is more like the bicycle shed in that sense and software still has an air of magic surrounding it to some people..
"mood, sincerity, and commitmentâ.
The difference between "hard science" and "others"? A classic gimmick that relies on feeling good, and may well work in marketing, sales, or even Human Resources. In the harsh world where folks have to build something that works - this "project management mindset" will fail. Period.
I have seen many, many sincere, motivated, and talented people fail. There is no substitute for a good, well thought out plan. Above all - brutal honesty, with your team, and with yourself.
-cluge
"Science is about ego as much as it is about discovery and truth " - I said it, so sue me.
Would you please post the links using the coral cache?
How hard could it be to either do it manually, or automate converting all links to be appended by nyud.net:8080 ???
You'd save many companies a load of bandwidth, and actually make good use of the coral cache...
Mod points are a dangerous tool. Abuse them wisely.
There are other effects, similar to that, in the US and elsewhere in the world. Consider Africa, where a confluence of war, famine, and AIDS have basically decimated a continent.
Entrenched poverty, and related gang and drug activity has combined with discrimination or bias in the judicial system in the US with the result being so many black men are in prison that it's affecting the demographics of the Republican party's most sacred institution, marriage. Of course, they are so concerned about preventing gay men and women from participating in the sanctity of marriage, whatever that means, that they haven't noticed so many of our nation's daughters will fail to follow God's sacred instructions to be fruitful and multiply, within the sanctity of marriage.
Consider that lead exposure, and probably mercury exposure, too, knock IQ points off the potential of those exposed. How much better off would this nation be if we hadn't exposed generations of kids (including the generations holding office today) to lead exposure? Hey, you Baby Boomers, do you think your kids are smarter than you? Well, guess what, they are. Kids under the age of about 25 or 30 in this country didn't grow up breathing lead fumes. If we would stop spewing mercury all over the atmosphere and oceans we could probably get another five points.
If you mod me down, I shall become more powerful than you could possibly imagine.
"Many of the people involved in FUBAR - developers, testers, team leads, managers - lack the qualifications, insight, or experience to make FUBAR a success."
..
Later on he says
"The FUBAR project is represented as something that has never been done before, and the staff as a world-class development team."
So which one is it McCain?
-Skeeterbug
Sadly, the responses noted would be somewhat common. Non sequiturs included. Most managers do not understand the value of systems architects (who should be responsible for the architecture of the integration of many complex systems) nor software architects (who should be responsible for the technical implementation details. As I'm sure you know, Bruce, these decisions are often made for political reasons, by people who don't have time to even hear about the consequences of different choices, who then essentially require their team to justify the decisions post-facto. In fact, this model is rather more common than any manner of objective careful reflection. Certain technologies never would have reached their current level of dominance, had IT decisions been made by people who understood exactly how much more expensive those decisions would turn out to be.
If you mod me down, I shall become more powerful than you could possibly imagine.
I agree these types of observations are often dismissed by management. They are not, however, very often debunked, which implies that they were incorrect, and proven to be so by a connected series of statements intended to establish a proposition.
There is an interesting problem with trying to persuade management of a problem like a giant failing project. They usually have a vested interest in the project success, and often the team. On the one hand they will dismiss arguments with insufficient detail. On the other, they will ignore or dismiss as "pedantic" or "overly detailed" any argument supported solidly by evidence. More often, however, they bog the bearers of bad news down with a giant pile of pointless assignments: "go produce document which explains "code fragility" and documents the extent of code fragility in this project, using industry standard metrics. " The point is, this debate cannot be won without going above the level of management that is causing the problem, up to the level of management which actually gives a hoot about whether the business objectives can be met. Doing so will probably cost the whistleblower their job, or their career.
If you mod me down, I shall become more powerful than you could possibly imagine.
This person is clearly poking fun at the Slashdot libertarian. Obviously they are aware that this type of spending is entirely equivalent as an economic stimulus, for a failed corporate IT project, and a failed government IT project.
If you mod me down, I shall become more powerful than you could possibly imagine.
TFA describes a project in complete disarray and a failure on all level. Despite this level of failure, TFA is pretty much anecdotal. TFA has no hard metrics or any other solid data to support the anecdotal conclusions. It is very possible it is accurate, but if a project is a complete mess, it should be easy to get good solid metrics that document this.
The fact that the author of TFA has none of this makes his assessment not only worthless, but places his work in the same category as the mess he was assessing.
If you need to slaughter a project, you need to:
a. Have a measurable set of goals for the project
b. Measure the current status against the goals
c. conclude status.
It would suffice to give a list of the different functions with number of showstopping defects for each function together with a description of the most glaring defect in each of the functions. Just pulled out of the bug tracking system verbatim would do. Example of glaring defect: "When the user push the OK button on screen XYZ, the server crashes, and all the data is lost. There is no workaround"
don't cut it off www.mgmbill.org
This sounds no different from the Videophone 2.0 project at Intel that I interned at... the EATS testing was abysmal and the incorporation of new bugfixes was almost a joke. It was no wonder that Dell, HP, Packard Bell, and other corp's abandoned the project in 1997...
Zaxyond@yahoo.com
I had to run errands for an hour or so and came back to discover that my server appeared to be completely crashed and had been so for over an hour. It's back up now. ..bruce..
Bruce F. Webster (brucefwebster.com)
We have a lovely park here, which was supposed to be a major bridge. The problem was, they hired two different contractors to build the bridge, and when the met in the middle, one side was about a foot higher than the other, making it useless. Rather than redoing enough of the work to make the bridge usable, they *abandoned* it and made it into a (rather lovely) park.
And the coast guard has abandoned an entire class of ships due to hull deficiencies. These now sit in a dock in Baltimore unused while someone figures out what to do with them.
"Be grateful for what you have. You may never know when you may lose it."
What's that say for Windows, then?
Granted, it's massively more difficult to reverse engineer a product - or re-implement its functionality to create a work-alike - than it is to design and implement something without those pre-existing requirements. But it at least gives some perspective into what kind of environment the Windows developers have to exist, what with the obscene number of applications which have to work - and not break - in Windows.
~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
damn_registrars said:
:P
"Anyone else find it ironic that this story about runaway development projects came right after the story on the release of wine 1.0?"
I wouldn't consider Wine itself to be a runaway project; actually, the runaway project is the system it's trying to be compatible with.
You gotta give the Wine guys props. They've pulled a proverbial rabbit out of a hat.
Codifex Maximus ~ In search of... a shorter sig.
I doubt it's either but those were two fairly recent big-time boondogles.
Never have I seen a topic about some guy (you) and have that same guy (you) post so often in the topic about some guy (you). It's like, you're taking all this personally. I found it rah'thu boring, ol' chap. Hans Reiser, we miss you !! Not so much the killing, but the audacity of killing and thinking to get away with it even when leaving all those clues.
the project was doomed from the start.
Actually, the PHBs will gladly ignore all input and advice from the lower-rung employees in their company, but will pay close, almost religious attention to the expensive consultant's reccomendations.
The reason? They believe that they get what they pay for. If it's expensive, it must be good.
In consulting work, never try to charge less, but charge more. You get a LOT more work that way, and it becomes less frustrating. PHB types hate having really costly consultant time wasted.
Funny, we saw the same at Dell Engineering around the same time. Some were honestly documenting , others were fudging. No one cared to verify the documented process. So much for stupidity and tons of time wasted documenting that could have gone for real product development. But Dell has a fat middle management layer that micromanages everything- if they could they would try to document how you shit too!
This memo of bf webster's is like a deja-vu on several memos I've written for several Big Firms containing their own Bob Winsoms. I think Webster wrote this memo with FUBAR and BigFirm so he can do a global search and replace for the next BigFirm he consults for. Saves a lot of time.
Sir, it is not ironic at all, unless Alanis Morissette is your English tutor.
You know, I honestly used to agree with you on the "preserve the old dictionary meanings of words" bandwagon and got annoyed when people "mis-used" words like irony.
However, a couple of years ago I took the stick out of my ass and realized that language is constantly evolving. Like it or not, popular (mis)use has expanded the definition of "irony". Is that really SO bad?
Need I bring up the example that "nice" used to mean "foolish", but common use changed its meaning? Did I miss the memo about the English language having achieved perfection and no further linguistic evolution was to be permitted?
Knowledge != Intelligence
That's part of changing the English language.
Everyone gets a say in it. And part of that "say" is saying someone else is misusing it.
Sometimes I think it's just gay. Damn straight gay it is.
That said, it helps to keep trying to maintain enough standard meanings otherwise we might as well be speaking completely different languages, and never get anything done (Tower of Babel).
It is infinitely more difficult to reproduce *exactly*. Even if you put MS developers (even the same ones - in the same sequence as they joined and left the company), you would not end up with exactly the same windows.
Even more so when you have to reproduce it in the middle layer. With both sides not under your control. see ReactOS, where they chose to not be the interface but just replicate. I would imagine it is easier for the ReactOS folks than for the Wine folks.
> actually, the runaway project is the system it's trying to be compatible with. :P
So it is *trying* to be a runaway project - otherwise it wouldn't be replicated exactly.
But if this is how you word a document of this importance, I'm not surprised it was ignored. Now, this might be written perfectly for an audience of one - and if that's the only person who read it, goody. Of course, if that's the case, it's unlikely to lead to wide spread changes. And I'm not defending those people who would read this, see the disgust and anger in it, and thus ignore it - but hell man, being professional does include being polite, writing for your audience, and being as objective as hell. Show object, factual data that you are basing your carefully considered opinions on. I've enjoyed reading through the document, and if I was on the project, myself, I'd be rooting for it - but I also wouldn't have a lot of respect for the professionalism of the writer, and I'd feel that, if the writer was specifically brought in to produce such a document, someone with more experience would have been nice. It certainly makes it more difficult to accept the opinions presented, when the presentation is lacking.
capital punishment is in eliminating murder
You make some good points, but capital punishment is not about eliminating murder, since we all know that it is impossible to do so. It is about reducing the occurence of murder, by making it more unpleasant, and removing murderers from society. It is also a good method of population control, that perhaps needs to be revisited in the future. I wonder how many people could be fed/medicated in the third world if we spent the money we use on housing murderers to give to the poor.
I find it interesting that your manager gets so upset by the mention of 'political reasons'. In my experience, internal office politics is usually the number one reason people ignore all the planning, common sense and warning signs that could help the project be completed, and yet your manager denies its existence. But yes, let's stay professional and ignore the huge elephant in the room.
And testers and designers. Basically every bad project I ever been involved in always was managed to death. Or rather, not so much managed to death as starved of actual work because if the managers actually managed the few people who do the actual work could get it done. Rather the abundance of managers keep the people who actual code/design/test from doing their bloody job. The problem is not unique to IT. I worked in a lot of different fields and seen it elsewhere, with more and more managers who come from outside the field they are supposed to manage you loose the awareness that the factory floor is where it happens. Simple example, warehouse. Double the order, so just work overtime. After all the suits do overtime no problem so why not the people in the warehouse? Because the suits start an hour late, have constant coffee breaks, 2 hour lunches, leave an hour early and only do overtime for an hour? The people in the warehouse start an hour early got only official breaks and do overtime for 4 hours doing back breaking work? It is a simple disconnect between the factory floor and the office and the same happens in IT except that it is even more hidden because all you see of a programmer/tester/designer is someone who is sitting behind a desk, so how hard can it be? I had one memorable project that was so bad it even reached the front page of a serious newspaper and had the company that bought the company it happened to take legal steps because they had been misled were at one point I was NOT just the only programmer but had also been lumbered with keeping the servers running despite having NO experience whatsoever with it. And when I complained I was told I was the one who had caused the delay by the person who had delayed the most by never ever getting anything done at all. (Needed a database dump from an old system, took 13 months) I learned my lesson since then, if the number of managers is greater then ONE, RUN LIKE HELL! Dilbert is wrong. it shows just one pointy haired boss and 4 techies. The reality is a dozen pointy haired bosses and just one techie wondering when he will get around to do some actual work between all the meetings and morale boosting excursions. I don't think you can fix the problem because the people who have to manage the managers are managers themselves. They CAN'T see the problem, because if they could they would only have one option and that is to fire themselves for having let things out of control and no manager would EVER do that.
MMO Quests are like orgasms:
You may solo them, I prefer them in a group.
No, it's incompleteness as well ;-) Lack of support for some recent popular subsystems like .NET 2.0, a lot of stuff produced with VC++8, etc. But older Windows crapware works surprisingly well on it.
With XP sticking around for years, Windows has become less of a moving target as well.
http://rocknerd.co.uk
You running WP Cache? It's extremely nice and very efficient.
http://rocknerd.co.uk
Rich And Stupid is not so bad as Working For Rich And Stupid.
Until the GC kicks in and causes your server side Java code to freeze for 10s of seconds (or even minutes) at a time while it does its thing.
Only an idiot does not exert control over what the GC does server side, when it's so easy to profile behavior and there are so many ways to tune GC (including disablement if need be).
Read up on modern VM GC tuning before you venture to put on that dunce cap again.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Yes, it may be that there is simply no solution at all. I sure can't see one...
"Not an actor, but he plays one on TV."
As Demming put it "the worker works *in* the environment, the manager works *on* the environment".
The worker can only achieve as much as management will let him/her. That includes hiring the right people for the job.
Instead of asking why software or IT projects are so crappy, we should be asking why is IT and software development management is so crappy.
putting the 'B' in LGBTQ+
Interesting, I just searched through all the contents in this Failed Project thread, for the keywords "Fannie Mae," "800 Million Dollars," "Core Project," "Epic Failure," "Monopoly," "Housing Bubble," "Recession" and "Fannie Mae," oh and "shithole," and found nothing here. Interesting...
Bruce F. Webster (brucefwebster.com)
Try telling that to us that have double-clicked on $neat_java_app and had to wait 30 seconds for it to load.
Try reading my post - from the start I was taking about server side where you have complete control over VM parameters.
Either you didn't notice that or you're just trying to dig your way out of your position enough to save face. I care not.
If Java is useless as a language to write quick and dirty little utilities to do things, why bother learning it at all?
Because millions of poeople/companies use it, and it's way more productive in server side development than C will ever be. Happy to help you.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Like it or not, popular (mis)use has expanded the definition of "irony". Is that really SO bad? Sorry, "expanding the definition of a word" is not the same as "not knowing what the hell one is talking about". Perhaps if we'd all just quack like ducks there would be world peace.
So does that mean that in your world people who say "nice" to mean "kind and pleasant" don't know what the hell they are talking about? (It used to mean "foolish and simple-minded") Are people who use the term "gay" to mean "homosexual" similarly confused? (used to mean "happy and carefree")
At what point does a popular alternate definition become "acceptable"? Is it when the editors of Webster's Dictionary finally get around to printing it, or is it when a majority of native speakers use and understand it?
The fact that our language is evolving and expanding is what makes it DIFFERENT from duckspeak. If you let it stagnate, we might as well just quack.
I believe that the initial poster who "mis-used" irony did, in fact, know what he was talking about. Additionally, I knew what he was talking about, and I'm willing to bet that 99% of fluent English speakers would know what he was talking about. Apparently you didn't, because you either missed or simply refuse to acknowledge the alternate definition update. Please try to keep up.
Knowledge != Intelligence
> So it is *trying* to be a runaway project
Hence, the 15 years.
Codifex Maximus ~ In search of... a shorter sig.