Slashdot Mirror
Struggling With Major IT Projects
Ant writes "This article discusses the poor track record of IT projects undertaken by the U.S. government, and says experts blame poor planning, rapid industry advances and the massive scope of some complex projects whose price tags can run into billions of dollars at U.S. agencies with tens of thousands of employees. 'There are very few success stories,' said Paul Brubaker, former deputy chief information officer (CIO) at the Pentagon. 'Failures are very common, and they've been common for a long time.'... Seen on Blue's News."
The summary ignores that corporations are just as bad, but that corporations don't admit their problems. This is one area we find the Fed fessing up.
Just like the sports analogy, the coach needs to put the players in a position to win. Bad management = bad coach. Unfortunately the techies are always the ones to get blamed and get fired.
Out of curiosity, how do you define "succeeding"? Is it a failure when user satisfaction falls short of what had been hoped for, or only when the new project is scrapped entirely in favor of the old setup?
What I'm listening to now on Pandora...
Hit nail on head fits here.....
I would also say it's the management's fault entirely when the entire technical staff is saying that they should have dumped this POS and should have never started the project.
Example:
Where I work, we had a mainframe based system with some web enhancments added on. 99.9 percent of the whole thing was custom written with COBOL using very specific specs and used a non industry standard DB called DATACOM. It was developed over a 20 year period at least with parts of it being in service for almost that long. It was very stable and it worked and was able to serve the company well. Then, some management muckety muck who is not even there anymore, at the suggestion of the president, started looking for a entirely new system. One that the users would not be using a mainframe terminal session. One that was client server based in a period of time the mainframe was and still is at this point getting a resurgence. TGhis was about 6 years ago now. We are entering the third quarter with the new system and after only one quarter of production, we had to upgrad ethe production boxes spending another 500,000 to a million with implementation time taken into account. Now that the system is generally faster, users are mostly satisfied and the IT staff still isn't. Few examples:
One end of quarter process that took 30 minutes to an hour, running in real time with users still logged in to the system on the legacy mainframe. The new system? 2-3 hours or more and the system it's running against must be locked out from changes during the process because the realtime process can take DAYS!
Patches as delivered by the vendor regularly don't work. I don't mean that they don't do what they are supposed to do after you get them in....they just won't install. The version of the product we have is a hybrid of what it should be. It still has it's dependence on the older datatbase layer being there and thats just so it can translate it's programming calls to Oracle SQL calls. The systems course they provide to train sysadmins on the specifics on the application does not include about 60-80 percent of the new version of the product....and this product has been out for at least 4-5 YEARS.
We went from a mainframe to a system that has about 4 times of the power, yet the system STILL runs slower. This is even on VERY modern hardware and using 2GB fiber for the SAN. Sure, thigns are getting better, but this product we bought into absolutly blows, yet gartner (they suck anyway) and others STILL laud it as a good system We're not the ONLY ones to complain.
The deal here is, I would not be surprised if we stuck with it, but it sucks so bad I would not be surprised if they finally cannned it. We went from a very efficient system that if there was a problem with it, the staff knew EXACTLY what to do and now we have to make half a dozen phone calls to support...support that's only available for 8 hours a day on a system expected to run the company 24 hours a day. It could be weeks or months before we find a solution. The funny thing is, this thing was presented as software that would do everything, yet we had to write many custom modules and paid the vebndor to write several custom modules, non of witch worked on the INSTALL DAY! Even after doing post-install setup the program was delivered bug filled. We had a higher up in the vendor's company tell the programmers who are working on our part of the system and WROTE the majority of the old system that programming was TRIAL AND ERROR! WHOO!
And all of this was for the sake of dumping terminal screens in favor of nice purty client/server interfaces. We STILL are not to the point we were 6 years ago. We're closer today then we were when we initally launched, but was are so far off, it's going to take another 3 years and likely another hardware upgrade to accomplish what's needed.
Sometimes management should just leave well enough alone.
You should get scared if your manager starts bringing in books like the Tom Peter's Project '04 and Who Moved my Cheese,,,,
Gorkman
As a manager, I agree with the first part but disagree with the second. It is indeed a failure of management when the IT project fails. However, it's usually because management has lost sight of what the original objective was and/or was unable to commmunicate it to the people doing the work.
Yeah? Well I think you're overrated too.
My observation is that is a combination of territorial office politics, automating bad processes instead of fixing processes, and not learning from past projects.
The second is a case where there are all kinds of intertangled, unnecessarily complicated business rules that are required or requested. Often these are dictated by legislation or attempts to "satisfy all stake-holders".
There should be some kind of bidding process on features such that features which gum-up the works will be charged to the customer somehow. Perhaps have a cost/benefit analysis/estimate be done on each feature, and chop off the ones that rank low (by being either too low priority or too costly).
Another thing I find totally lacking is any documentation of the design decisions. Before spending gazillion dollars on a fat project, the design and architecture should be seen and/or suggested by several expert eyes and every one of their written critiques and evaluations should be saved, whether used or not.
Then when a project succeeds or fails, one can see which ideas and/or which consultant/expert seemed to have the best insite or vision. Otherwise you keep reinventing the same mistakes over and over again.
Table-ized A.I.
It's not that the requirements are undefined, that is problem but not the major one. Every project I've worked on has had requirement changes.
The major problem is that management and techies do not fully comprehend what a lack of requirements means.
If open source is showing anything it is showing that release early release often is the way to go. Let the users pay with your system as early as possible and be prepared to change everything because it's doubtful they know what they want either.
Techies should be writing code that can be easily ported/changed/rebuilt/removed fluid software. The number of arguments I've had with other developers who want to build a system the one right way. Then they cry foul when the requirements change.
Managers need to get on board with the change culture. Requirments is at best a guess, don't go planning a release date based on them. If you have to implement on a fixed date then you'd better be ready to go live with a non-complete system. Plan for a release 2 etc.
Of course, we've known this since the 70s. I wonder how long before people realise that software should be configured not custom built.
With BlueGene, the US gov't approached IBM and told them "We want the fastest super computers in the world. We want to eventually reach the the Petaflop range. Here's some money. Do it" and IBM happily complied. Late last year, the BlueGene/L prototype recaptured the title of world's fastest computer from the Japanese. The BlueGene/C design is due (on time) in June and should be available from the foundry in August (full discloser - my grad work involves testing & verficiation of this). The lesson? Where IT is concerned, the Government has a legitimate interest in outsourcing it to reliable companies (prefarably US based for security reasons).
To make laws that man cannot, and will not obey, serves to bring all law into contempt.
--E.C. Stanton
The Japanese software industry works like a factories. Not much innovation, mainly production. I'll give them credit for their quality though. They produce way less bugs than western or Indian programmers for that matter. Read Michael A. Cusumano's book Japan's Software Factories: A Challenge to U.S. Management. Google has more.
On of the classic big software projects gone bad was the failure of the revamp of the air traffic control system, which was stiched together with creaking mainframes and had old CRT terminals. The rewrite/redo/new system was a classic "Bridge too Far" (Operation Market Garden from WW-II a classic "massive rewrite" instead of an incremental approach. They were going to use paratroopers to capture a whole lot of bridges in Holland, and then the main body of the army was going to come motoring through all of those bridges and straight into Germany to end the war. The main army got bogged down at the first bridge and the British 2nd Airborne Division got hung out to dry at the last bridge, and a British general offered up British understatement as to why the whole plan failed as "it was simply a bridge too far."
In the case of the ATC rewrite, they concentrated on the ATC controller workstation -- the thing with the big CRT where all the flights show up on radar but with alphanumeric info added -- but it was going to be the ATC workstation to end all workstations, and it got caught up in a mire of specs.
I surmise that it was supposed to be a massive bitmap graphic CRT back in the day when megapixel bitmap graphics was bleeding edge, and it was supposed to customized, both for the preferences of the controller as well as the geography it was to cover using some kind of configurable software, back in the day when people hadn't heard of OO and GUI frameworks.
I always thought they should have done something incremental, but I guess I never understood their problem domain well enough to know what incremental would be.
But what ever happened to ATC? Are they still using battleship CRT monitors driven by creaking mainframes? Did they come up with something that works using newer technology?
Q. How do you make sure you're solving the right problem?
;-) ), and this and that.
A. If GM builds a the wrong car for the wrong market are you going to blame it on anyone but GM leadership? If not, then why are you going to give them a pass on implimenting a network infrastructure that fails so meet their needs?
Excellent point! Somehow when things fail in a private company, it is always the fault of the "suits", while when things fail in the Gov't -- well, it was just too tough, and not enough funded (please increase my tax rate!
Comparing the waste of the modern Big Gov't to the reasonably (if only sometime inefficiently) run corporation can be eye-opening, but most of the people here would prefer to close their eyes shut for whatever political reasons.
Paul B.
Interestingly, she was also an IT supervisor at the White House during the Clinton years and she actually threatened people under her with jail time if they said anything about that coding error that resulted in those 100K emails subpoenaed for the Lewinski fiasco. It's amazing how such fools can not only fail to execute their job duties properly, but also actually achieve higher positions at the same time! Ah, the wonders of large, inscrutable government bureaucracy.
Judging by some of the PhD's I've met, though, it very well COULD be difficult to tell the difference between "pompous jackass covering for a lack of education" and "pompous jackass who thinks they know everything, despite the narrow focus of their PhD".
If a job's not worth doing, it's not worth doing right.
So you're telling me that you've never seen a situation where the technical staff overestimates their own ability, and underestimate the potential for failure?
No. The closest to that is when the technical staff made a recommendation based on the data available from the vendor, but the vendor was over-hyping the product. I've worked one place where I was asked "will this work" and answered "no." So, since the management must remain blameless, they hired a consultant to come in and declare it will work. Then they required me to impliment it in the way that the outside consultant laid out. It didn't work, but at least the techies weren't blamed, even when they were set up to fail. Oh, and we still use the same outside consultants all the time to get answers we want to hear, even though they are wrong more than right. We could flip a coin and save a few $100k per year.
It would be nice if problems were just techies that were too insecure to declare that they couldn't do it. Those are easy problems to fix. It's when the entire plan is bad that there is often nothing that can be done for it.
Learn to love Alaska
Back at Rockwell-Collins, there was this one project that I was on for which the history of it was... well, lets just say that they didn't like to talk about it much. Apparently, one engineer had worked on the project for something like a year and a half before they realized that he hadn't been doing anything when at work, had been forging his progress reports, and all he had to show for the project was an animation of a dancing frog.
We also have a halon fire extinguisher. Its always nice to have a fire extinguisher that kills people around.
We have the same problem in the UK, and I expect it is the same everywhere. I have worked on good projects backed by government money, but the problems seem to arise when the government is the provider of the detailed requirements. I think this is because the government can only provide top level requirements, and the detail should be fleshed out by experienced requirements analysts. Unfortunately, government organisations do not consist of high achievers who have made their way to the top through good decisions. Government organisations consist of people who have made their way to the top by cautiously waiting long enough, and their natural instinct is to avoid tough decisions. Of course, tough trade-offs have to be faced and dealt with to reach a coherent set of requirements. Any project with the government will not be able to do this without a lot of futile hand wringing and back tracking. Coupled with all that is the fact that government people have an innate belief that they, not the technologists, are in control. They believe they can realise change faster that the technologists can provide it. Technology development has a pace that technologists understand, while government people have different objectives that may be changed in a hurry as public opinion sways this way and that.
I stole this