Slashdot Mirror
Struggling With Major IT Projects
Ant writes "This article discusses the poor track record of IT projects undertaken by the U.S. government, and says experts blame poor planning, rapid industry advances and the massive scope of some complex projects whose price tags can run into billions of dollars at U.S. agencies with tens of thousands of employees. 'There are very few success stories,' said Paul Brubaker, former deputy chief information officer (CIO) at the Pentagon. 'Failures are very common, and they've been common for a long time.'... Seen on Blue's News."
Major IT projects touch a lot of people. If you can't get everyone on board then the project is going to be very tough to complete succesfully. For that reason, the only real blame for "most" IT projects failing is leadership problems.
It's harsh, but true. If these agencies had had better leadership and management, the projects would have been delivered, or at least never started in favor of something better. Blaming is on anything else is an excersise in passing the buck.
The only IT projects that failed that I know are the ones that have bad managers... Most of the time that means someone who doesn't listen to the people that do the actual work but there are other reasons...
Peter.
I can only agree that very few large IT projects are succeeding. I put the blame partly on managers in charge of the project that are too non-technical and distant from the nuts and bolts of what is going on. They push the freight train on with the theory that the project can be brought in through determination and hard work. It can't. It has to be brought in by clever people who know what they are doing. And these manager types will push the train on till it goes over the cliff when better people would have known much earlier that the bridge was out.
The summary ignores that corporations are just as bad, but that corporations don't admit their problems. This is one area we find the Fed fessing up.
Click here.
Ant(Dude) @ Quality Foraged Links (AQFL.net) & The Ant Farm (antfarm.ma.cx / antfarm.home.dhs.org).
The basic idea of an "IT project" is to implement something that has never been implemented before, or to replace something wholesale.
This flies in the face of every software engineering textbook. Software, like flora, grows in its environment. Trying to introduce something brand new into an ecosystem is asking for widescale decimation of existing services as well as the increased likelihood of the introduced-species death.
So the key to getting working "IT projects" to succeed is to build on past successes. It's never the "Start from nothing, plan, implement" projects that do well. These typically go way over budget and way past the deadlines. It is the little "I need a little tool" projects that start off small and then are brought together or have extra features added to them that succeed.
Look at your bank's ATM system. When those machines first arrived, they didn't do half of what they can do now. It was through a gradual building upon what works and weeding out what doesn't that allows us the ease of personal banking today. Same with any system, even Linux. Linux started out as a small project to implement a Unix-like kernel. Now it is a huge business and the project itself is much larger in scope than the original idea of Torvald's.
Improvement, not creation, is the key to successful projects.
When a deputy CIO of the Dept. of Labor and than Homeland Security Department has bogus degrees and has never been officially questioned about her educational experience (or lack thereof) for years, its not hard to see how gov't IT could be atrociously run.
From other articles about her, she was notorious in promoting her cronies, many of whom were also incompetent while passing over for promotion and bonuses those who knew what they were doing. Apparently Laura Callahan had a reputation for going ballistic when the occasional techie caught on to her and questioned some of her decisions. In hindsight, its rather obvious why she was so insecure.
Most of my development experience tells me that IT development failures almost always are due to a lack of defined requirements. People *think* they know what they want, but they usually don't. And what they do know, they often cannot articulate. Kinda like the Supreme Court's view of pornography "I can't define it, but I know it when I see it."
Most of us in the Development / Software Engineering fields see this on a regular occurrance. The same thing is probably happening here.
The reason these projects are failures, or cost too much is because they are not being done out of need, but from strings pulled to dole out corporate welfare. Every industry the US is internationally competitive in (except maybe Hollywood) has (or had) most of it's R&D paid for by Uncle Sam - aerospace, the Internet, pharmaceutical companies and so forth. It's the old Keynesian thing of the government burying bills in old wine bottles and having some company come and dig them up. Government spending, which in the US usually means Pentagon spending, has been greasing the wheels of the US economy since FDR took office. The only difference between the two major parties is Republicans tend to want to build rockets/lasers that can shoot down rockets and that sort of thing, while Democrats want the money to go towards biotechnology and things like that. If you want to see what's going on, don't look at the end result and try to discern what went wrong, but look at the legislative process, and what pressures are in effect there. Billions of dollars was not really wasted - it made work for many people, imagine what unemployment would have been if it hadn't. It's the old bills in buried wine bottles story. I mean think of some of the ridiculous things proposed - billions for a "missile-defense shield"? It's just a way to spread money around. I don't like how the Democrats or Republicans do this, I have other ideas of how that money could be used for make-work.
I think projects of that scope should stage such large developments, start with a general specification for the system and the desired end result and interoporability, then develop and roll-out modules progressively. As you debug the core modules and define the additions you can tune/revamp as you go. Then when you have a an unexpected problem with your thousand clients you are only dealing with a portion of the functionality not some monolithic spaghetti code. By the time you get to the end of the development you are working on a field-hardened platform.
Would it take longer or cost more? - well given the time and cost overruns of many of these big projects it might be more economical if not mre timely.
"Enjoy what you're doing! If it becomes drudgery, you're doing it wrong!" - Jim Butterfield
'Failures are very common, and they've been common for a long time.'
They aren't really failures. They are deliberate government corruption. Anything that has been "common for a long time", with no effective corrective action, is deliberate.
The IT corruption is small compared to the military procurement corruption, where the dishonesty can be kept more secret. The U.S. government is the world leader in buying equipment to kill people and destroy their property. (The least socially skilled way of relating to other people is killing them.)
U.S. citizens, it's 7 PM. Do you know what your government is doing? Unfortunately, you don't.
I thoroughly agree. I'm a project manager that used to be a tech. I become a pm because I'd worked under too many of them that had *no* idea.
:) These lessons came to me through the development of a fair amount of scar tissue. These days I never employ a project manager that doesn't burst into tears when asked about their worst project.
Projects are all about scope. Defining what it is that you're doing. Everone thinks that's bleedingly obvious... and they're right. But it ain't easy to do.
Once you're got the scope, the rest should be easy. But isn't. Another classic big project blunder is the lack of realistic funding and schedule. Nobody want's to say it's going to take megabucks and go for years.
So instead you end up with "it won't cost much or take very long". Guess what... budgets and schedules "blow out"! More likely they take as much and as long as someone who understood the aforementioned scope would have said in the first place.
Even when the basics are followed things go wrong. This is the final in the classic series of blunders. If something is starting to look bad - don't tell the project sponsor... we'll be right... maybe.
Big no no! Tell the project sponsor *now*! What's wrong, why it went wrong and how you intend to fix it!! You'll get more respect and less stress. Both of which make it more likely you'll get it sorted.
Ahhhh. I feel better now
Forget the truth. Science is fact.
My observation is that is a combination of territorial office politics, automating bad processes instead of fixing processes, and not learning from past projects.
The second is a case where there are all kinds of intertangled, unnecessarily complicated business rules that are required or requested. Often these are dictated by legislation or attempts to "satisfy all stake-holders".
There should be some kind of bidding process on features such that features which gum-up the works will be charged to the customer somehow. Perhaps have a cost/benefit analysis/estimate be done on each feature, and chop off the ones that rank low (by being either too low priority or too costly).
Another thing I find totally lacking is any documentation of the design decisions. Before spending gazillion dollars on a fat project, the design and architecture should be seen and/or suggested by several expert eyes and every one of their written critiques and evaluations should be saved, whether used or not.
Then when a project succeeds or fails, one can see which ideas and/or which consultant/expert seemed to have the best insite or vision. Otherwise you keep reinventing the same mistakes over and over again.
Table-ized A.I.
You've obviously never worked on a government program. Sometimes the oversight is all there is and the project is just secondary.
It's simple: I demand prosecution for torture.
With BlueGene, the US gov't approached IBM and told them "We want the fastest super computers in the world. We want to eventually reach the the Petaflop range. Here's some money. Do it" and IBM happily complied. Late last year, the BlueGene/L prototype recaptured the title of world's fastest computer from the Japanese. The BlueGene/C design is due (on time) in June and should be available from the foundry in August (full discloser - my grad work involves testing & verficiation of this). The lesson? Where IT is concerned, the Government has a legitimate interest in outsourcing it to reliable companies (prefarably US based for security reasons).
To make laws that man cannot, and will not obey, serves to bring all law into contempt.
--E.C. Stanton
Mismanagement. Same as large projects in commercial organisations.
;)
Basically, layers of layers of people produced by a hierarchical system that encourages mediocrity in ability and excellence in deception. Liars with superb brown-nosing skills and minimal ability leading skilled developers without those career-climbing skills. Recommendations from the knowledgeable ignored for reasons as petty as favouritism and wounded pride. Uninformed directives passed down. Valuable feedback distorted or disregarded going upwards. Career employees who couldn't care less doing the minimum they can and maximising the amount they seem to be doing. Burnt-out and defeated employees who believed once but can no longer care. Code and credit theft. Incompetents hiding their errors by sabotaging the work of others. Narcissistic managers who simply don't want negative feedback. Accomplishments delayed or distorted to fit the drip-feed system of delusional managers. Sacrifices of the innocent to protect the careers of the guilty. All held in place by a system that encourages all the negative aspects and hides it away in a nice, neat, convoluted bundle.
Did I miss anything?
And people are surprised that large projects frequently fail.
Some projects fail because they don't have enough resources, or not enough time has been allotted. Sometimes, the project planners choose the wrong implementation or try to solve the wrong problem. Sometimes, projects fail because the users don't really tell you what they want.
Are all of these 'leadership problems'? Sure, you can blame the leader of the project (or his leader) for those problems - after all, they should have seen them and fixed them.
But then all you've done is group a lot of problems together under "who to blame" and not tackled the harder problem of avoiding those pitfalls. So while I agree that leaders should stop projects from failing, the root causes of the failures are far more complex than just "leadership problems".
-Peter
Some projects fail because they don't have enough resources, or not enough time has been allotted. Sometimes, the project planners choose the wrong implementation or try to solve the wrong problem. Sometimes, projects fail because the users don't really tell you what they want
Amount of resources and time allotted are both directly related to leadership. Leaders decide both of these things. Choosing the wrong implimentation could go to the tech folks, but solving the wrong problem is most certainly a leadership decision too.
I'm not trying to lump every decision on the leader and I'm not trying to say others can't screw up, but the cold hard facts are that projects that reach a certain size tend to fail because of things leaders should be taking care of.
Specifically:
Q. Who dicides how much time a project has?
A. Management.
But... what if the underlings give management a false impression about how long a task should take?
A2 This is management 101: Work with what you know about your people ans statistics about past projects to determine if they're proposing adequate time. It's TOTALLY a management failure if the amount of time given a project is wrong.
Q. Who decides how much resources should be thrown at a project.
A. Once again, this is TOTALLY a management decision. Yes your underlings may give you incorrect data to base this on, but if you're consistantly getting BAD DATA then it is a leadership FAILURE if you continue to believe the bad data.
Q. How do you make sure you're solving the right problem?
A. If GM builds a the wrong car for the wrong market are you going to blame it on anyone but GM leadership? If not, then why are you going to give them a pass on implimenting a network infrastructure that fails so meet their needs?
Leaders solve the macro problems of the company. Large IT projects are part of this solution. A large IT project is about as complex as building a new HQ building. Leadership does not allow new HQ buildings projects to fail. Why on earth are you letting them get away with not managing IT projects (infrastructure by defininition)with as much dilligence?
Several times I've worked on projects where "the people doing the actual work" (i.e. techies) have been responsible for the ultimate failure; they've been given too much authority and made decisions that've ultimately sunk the project.
Examples:
- "we know it'll scale to 10k users, once we take the time to optimise it. We'll do that later" (project ultimately scaled to ~100-200 users max)
- "upgrading to the new version of tool XXX will let us solve lots of our problems straight away" (maybe so, but it added lots of new problems and dependency issues that blew the project out of the water)
- "we'll redo the crappy UI later" (not after you've made loads of incorrect assumptions about workflow based on a UI that already you knew was wrong, you won't!)
- "we've just attended a MS Web development seminar that told us our objects should be stateless, so that's what we'll do" (...to the point of having to verify e.g. user identity multiple times for each page loaded. This particular project brought down a country wide intranet when it was deployed, without prior testing because the developers thought "it wasn't necessary")
- "Bob's got that problem under control. We don't need to worry about it" (...until Bob, the single gun Tandem guy, left the company and we were left with a totally undocumented Tandem interface that nobody understood in the slightest)
- "that's OK; Microsoft are sending out a consultant to deal with that problem" (...sigh... Does no-one understand that the main job of a field MS consultant is to sell more software licences, not to fix problems?)
Every one of these issues was dealt with by a lead techo in the manner described above. Mgmt deferred to the lead techo in each case, and the projects suffered as a result.
Yep, I understand that you could call all these people "managers" rather than "techos", but each of these decisions was made on a supposedly well thought out technical basis. If these people are "management", then so are 90% of the people on Slashdot.
If we're going to characterise all management as PHBs, then why not also characterise all techos as:
- making incorrect assumptions, then extrapolating endlessly without attempting to verify the original assumptions
- assuming tools from Vendor X are golden (yep, I'm continually amazed how often this happens)
- relying on vendor X to provide a solution to problems when they occur, and not investigating workarounds in a timely fashion
- believing acceptable performance is always just a few code tweaks away
- assuming they know more about usability than designated subject matter experts
- endlessly reinvent wheels ("What? That object isn't a 100% fit for our problem? Better create a whole new one that works 1% differently and requires 1000 new lines of code to be maintained")
After 20+ years in the industry, I firmly believe that the best IT managers are those who have worked in multiple technical areas, as they can then see through the tech crap as well as the mgmt/project crap.
You know, I used to be like you - saying that the people in the trenches are the ones that make things happen, but I found I was wrong after starting a business. I bet you're currently in the trenches and have never been in upper management...
Without leadership, the people in the trenches would do whatever they wanted, anytime. They wouldn't have the right tools, they wouldn't conform to proper coding standards, they wouldn't be able to effectively meet with clients (think carefully on this one... Most groups will elect someone, aka: Leadership, to talk to the client.), there would also be budget issues, etc.
Trust me, I believe that the trench people (haha..) deserve a lot more credit than they usually get. The leadership though, needs to take care of the people in the trenches and thats what most businesses are missing these days. Hell, I worked for a company that provides a service to the government and I made enough to qualify for food stamps while management was driving around in BMW's and Porches - a classic Pointy Haired Boss company.
Its all about how you treat your employees. I've been fortunate enough (or unfortunate, depending on how you look at it) to work for a company that spoiled me when I was in the trenches. They made all of us at the bottom feel like we where the only ones that mattered and they where right. That company went profitable so fast and made so much money, it wasn't funny. Imagine being 19 years old and trying to figure out where you where going to park your Hummer H1 in 6 months while others are buying $200,000 houses with cash!
I've also worked on the other side of the fence for companies that are deeply rooted in the "Don't trust your employees" mentality - they happened to be an investment firm. They prided themselves on hiring the best of the best (top 1% that applied) but had an employee turnover ratio of over 26% a year... Oddly enough, they where very open about it in the interview and liked the fact that I brought it up. They felt it showed that I was truly interested in the long term... I bailed on them after 6 months for something better even though I was well paid.
Again, it IS about leadership. If leadership isn't doing their job, you don't like your job (or you just don't like working). Most geeks want to work for a cool company that works - lets use Google here... Why is Google so popular? Leadership.
Back at Rockwell-Collins, there was this one project that I was on for which the history of it was... well, lets just say that they didn't like to talk about it much. Apparently, one engineer had worked on the project for something like a year and a half before they realized that he hadn't been doing anything when at work, had been forging his progress reports, and all he had to show for the project was an animation of a dancing frog.
We also have a halon fire extinguisher. Its always nice to have a fire extinguisher that kills people around.
We have the same problem in the UK, and I expect it is the same everywhere. I have worked on good projects backed by government money, but the problems seem to arise when the government is the provider of the detailed requirements. I think this is because the government can only provide top level requirements, and the detail should be fleshed out by experienced requirements analysts. Unfortunately, government organisations do not consist of high achievers who have made their way to the top through good decisions. Government organisations consist of people who have made their way to the top by cautiously waiting long enough, and their natural instinct is to avoid tough decisions. Of course, tough trade-offs have to be faced and dealt with to reach a coherent set of requirements. Any project with the government will not be able to do this without a lot of futile hand wringing and back tracking. Coupled with all that is the fact that government people have an innate belief that they, not the technologists, are in control. They believe they can realise change faster that the technologists can provide it. Technology development has a pace that technologists understand, while government people have different objectives that may be changed in a hurry as public opinion sways this way and that.
I stole this
ERP implementation 1: At this place, there was something, let's call it the TPS report. It was supposed to be an automated replacement for a number of things that were manually generated from database extracts and number crunching. So, an interview with, oh, let's call her Suzy goes something like:
Me: so why don't you use the numbers from the computer?
Her: they are wrong, I use the spreadsheet from Fred instead.
Me: so what is wrong with the computer numbers?
Her: If I use the numbers from the computer, then the TPS Report will be wrong.
Me: so what is wrong with the numbers in the computer?
Her: well, the sales guys all get commissions based on what is in the computer, but that is not what actually got sold. Jeff places orders for customers, and the customers send it back for credit (called channel stuffing). So Jeff gets credit for the sales, but not penalized for the returns. George puts his numbers in 3 weeks late. And the VP, Ed, he just makes up numbers to meet the monthly quotas.
ERP implementation 2: At this place, the (mis)manager, let us call him Bob, decided that he wanted to keep 4 sets of books. Set 1 was the one that the IRS got to see. Set 2 was the one that the board of directors got to see. Set 3 was the one used to show to investors and the stock market. Set 4 was the real set of books. I told my boss that I wanted nothing to do with this Enron-wannabe. I won't work for SPECTRE, nor will I work for Enron. I had to quit to get away from the project. I understand they wasted over $8,000,000 trying to implement this evil piece of stuff. And they never understood why the numbers did not match. There is a solid reason for Sarbanes-Oxley, there are still a lot of companies who succeed at getting away with this sort of business.
As long as the people putting bogus numbers into the system had more political power than the people putting the computer system in place, the system was never going to work. Corruption and cronyism are not exclusively endemic to government. Most programmers are naive, they lack an understanding of the power that politics have over the way that work really happens. Stuff like the above 2 samples are why older programmers are cynical and sometimes bitter. Part of the reason that companies look for young programmers (under 30) is that those naive babes in the woods haven't been through the woodchipper yet.
If you hear of a "failed" implementation of some ERP/CRM system, dig deep enough and you will see things like the above samples.Only in America are people so naive as to say things like: let's leave politics out of this.
disclaimer: this is an encore posting.