Slashdot Mirror
6 IT Projects, $8 Billion Over Budget At Dept. of Defense
McGruber writes "The Federal Times has the stunning (but not surprising) news that a new audit found six Defense Department modernization projects to be a combined $8 billion — or 110 percent — over budget. The projects are also suffering from years-long schedule delays. In 1998, work began on the Army's Logistics Modernization Program (LMP). In April 2010, the General Accounting Office issued a report titled 'Actions Needed to Improve Implementation of the Army Logistics Modernization Program' about the status of LMP. LMP is now scheduled to be fully deployed in September 2016, 12 years later than originally scheduled, and 18 years after development first began! (Development of the oft-maligned Duke Nukem Forever only took 15 years.)"
Man someone should have told me that a long time ago, 8 billion is nothing to sneeze at.
It's nice to see someone in our government giving 110%.
First rule in government spending: Why build one when you can have none for twice the price?
if you specify the actual cost during the planning phase, then they wouldn't be started in the first place. So people make best case estimates and then reality strikes, the actual cost exceed the allocated budget.
...it looks like they originally expected to take 6 years to roll out their plan. Even if they'd been on schedule, by the time everything was in place, it would have been obsolete.
Pretty much everything the Pentagon does is over budget, behind schedule, and budget-wise, generally a spawn of wishful thinking. The "cheap" Littoral Combat Ships were sold to Congress as sub-$250 million craft. They're currently just under $700 million apiece. The "cheap" F-35 was promised to be no more than $60 million a copy or so. They're now just under $200 million a copy, flyaway (more expensive than the F-22 they were supposed to compliment). The new Ford class carriers... an evolutionary development of the current Nimitz class.... will now cost 2 1/2 times as much as the last Nimitz that was launched just a few years back.
Why should DOD software be any different than DOD hardware when it comes to wishful thinking from the brass?
Life is hard, and the world is cruel
That is normal for any successful (lower) government project.
I expect that the contractors were staffed with lots of "project planners" and "requirements specialists" who went straight from the service to work on these projects. And you can be sure that the ex-military are extremely unlikely to buck the system and stand up to uniformed types. And those in uniform know that they can climb on the retirement gravy train as long as they don't make life too hard for the contractors who they expect to work for when they get out.
It's a recipe for disaster. Nobody is going to make waves, because they are all too busy looking out for their common interest. It's another example of the endemic corruption that is steadily eroding the fundamentals of US society.
Of course this is small change compared to what goes on in the financial sector...
Why is Snark Required?
LMP is now scheduled to be fully deployed in September 2016, 12 years later than originally scheduled, and 18 years after development first began! (Development of the oft-maligned Duke Nukem Forever only took 15 years.)
Hey, they can still be ready before Elite IV.
Apparently the logistics of killing dirty little foreigners is quite complicated.
C, C++, PERL, PHP, Ruby...xyzzy
Refactoring to another language is expensive!
...both on the DOD side and on contractor side. But god forbid we cut their budget.
Government subsidizing private corporations, overruns,
non-transparency, corruption. And look at the straight
faces meanwhile. What a joke.
... the best that money can buy.
When government work used to be done by government, it could have gone over budget: you have to buy more raw materials or pay more people to work more hours.
But the whole point in outsourcing is that you pay a fixed amount to third parties to complete a specific job, and they take over the responsibility for making a profit (or, at worst, breaking even).
OK, I lie. The whole point in outsourcing is to give treasury money to your friends, and erode the state in favour of scrounging corporations.
Please tell me again why Java and C++ are so much better than COBOL and Pascal.
Please tell me again how an diverse collection of eager kids can replace a handful of bitter old white men and do it so much better, faster and cheaper.
I was with you in your first sentence.
You're second sentence however is racist & ageist (and perhaps sexist) drivel.
So wait a minute...if you're always forced to go with the lowest bidder this can happen? I would have thought the lowest bidder would have also been the most reliable and skilled. Who would have known?
Scope Creep!!!
Come on people. Say it with me.
Federal. Prison.
FEDERAL. PRISON.
Are we so inured to this that we can't even speak the words, let alone call our congresspeople? Will we not even push people to ask whether the next president will be calling for the prosecution and imprisonment of the people responsible for creating a billion dollar, 18 year "army logistics" software development project?
Don't give me that "mistaking malice for incompetence" bullshit. That's exactly what's wrong with this country. Just because it's computers, don't tell me you can't tell a $100 toilet seat when you see one. A couple years late may be incompetence, but you should have the FBI given all necessary clearances and set them crawling all over it. At 8 years into a 4 year project, you fire the buy-side project managers and cancel the project, whether you uncovered fraud or not. Fail to maintain even these basic standards, and no estimate in time or money is ever real, and every contract becomes open season for treasury looters. Oh wait, like it is today.
There is no way on earth or heaven that a logistics system can cost this much or take this long to build. And I would say those so corrupt or negligent as the ones running implementation at the vendor or running procurement within the military should be behind bars. This is not a joke, people - this is keeping American troops in a decaying and ancient logistics system so that some weasel can steal your tax money.
We could all start the backlash right here, today.
Tired of Political Trolls? Opt Out!
Apparently it is the sacred duty of governments to waste money, rather give it away to random corporations than ever run the risk of making a profit or hint at competing in any market. So it looks like this government department is doing a fine job, making its corporate chums filthy rich off of taxpayer money in the process. Redistribution of money was what taxes are all about, wasn't it?
Or at least, that seems a pretty accurate description of this country's government's actions over the years. Any economist reading care to refute this? Please?
The Navy's ERP system has already been "deployed" at certain commands such as NAVAIR and SPAWAR, so it isn't like they have spent this money and not produced a product. The system has issues, some of which are outright bugs (e.g. losing employee time sheets), some are poor design choices (e.g. the purchase request part of the system had issues with $0 line items on orders, previously we had to put in $0.01 to make it work), and some are dumb policy choices by the DoN or the local command (e.g. we are not allowed to use the built in leave request system, we still pass around e-mails or signed PDF documents).
Overall it is an improvement over the systems it replaced - for me that was a series of in house systems that essentially emulated the original WANG environment and re-used COBOL code. It does make it more difficult to fix things and I still believe the system has a hard time working with the DoD's "accounting" system i.e. different colors of money, lines of accounting, money that expires, etc.
/* ICBM Coordinates 32.78N, 79.93W */
There's engineering solutions and military solutions.
Engineering solutions have known goals (don't laugh, I'm writing about an ideal situation here).
Military solutions cover contingencies which may shift over time. They are also highly vunerable to political whim, up to and including outright bribery.
Obama 2012!
From TFA: "The department is racing to meet a statutory September 2017 deadline for passing a full financial audit."
Ya gotta love it. Any publically traded company has its accounts audited annually. The government is so out of control that it looks unlikely to meet a deadline of a successful audit five years in the future.
The government ought to be required to follow the same standards required of companies. No one has any idea what the financial status of the US government really is, least of all the government itself...
Enjoy life! This is not a dress rehearsal.
Granted, just anecdotal.
Generally, the contractor will start looking and like acting the customer, especially if you only have one customer.
Also, technical projects are hard. Requirements can change by a lot. And in some segments of the military government officials are only in their positions 2-3 years, so they don't get the level of expertise they need.
Except for ending slavery, the Nazis, communism, & securing American independence, war has never solved anything.
Pretty much everything the Pentagon does is over budget, behind schedule, and budget-wise, generally a spawn of wishful thinking
And by "wishful thinking" you mean the wish that a whole bunch of pork will land in a barrel somewhere, right?
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
I use Navy ERP (the Navy program to help us be audit ready).
It definitely looks like it was designed 10 years ago, and it feels like it when you're using it as well. It's slow and has a clunky interface... but it does work and all the records you'd want for an audit are there.
It may not be "fully deployed", but it has been used in the Navy IT acquisition labs and offices for several years now. I really have no idea whether other people aren't using it and why they wouldn't (ok, that's not true, I do know why: some people hate the idea of being audited and are pushing back on ANY oversight which may catch them spending money inappropriately... there goes any hope of signing this comment).
I'm already looking forward to the next version.
When you spend 10 digits on software you should come out with a univeral simulator or a Matrix-like sex game. Anything else is unacceptable.
"When information is power, privacy is freedom" - Jah-Wren Ryel
It's the Department of "Defense". I would be surprised if it wasn't over budget. (I'd also be surprised if there wasn't some pork being doled out to contractors in key congressional districts from these programs.)
Amakudari. And it is a recipe for disaster in every industry that it occurs in. . .
Sdelat' Ameriku velikoy Snova!
Yes, but ... that is a response to the miserable failure that the project already is. The fundamental problem, is "If there's one thing every junior consultant needs to have injected into their head with a heavy duty 2500 RPM DeWalt Drill, it's this: Customers Don't Know What They Want. Stop Expecting Customers to Know What They Want. It's just never going to happen. Get over it." --Joel Spolsky.
The customers, in the DOD's case, are combat leaders, not program managers. They don't know what they want. They know what they don't like about what they have, and they think they want whatever vendor presentation is given to them. Then, add 5 layers of committee in the JCIDS process, and it can never work.
The system is fucked, but not substantially worse than in the rest of industry for large software projects.
Seems like a lot of money, but this is probably the most sophisticated systems of it's type every created. I would imagine there is somewhere in the 100's of thousands of unique requirements to be tested. Dozens of allowed hardware configuration, distributed data, and tens of thousands of users.
Big Job. Big Money. When you are doing something big, and new, it takes a long time.
Like when you had to recode your php home page for an update.
In the USA, I know, CIO/CTO that are highly certified, but frequently totally unqualified. Yes, there are many good/great C*O/managers in .Gov, .Mil, .Com and far too many hubris-clowns with exceptional social-engineering skills [They seam to breed like rabbits (in some locations) and feed from big-rabbits' pellets dispenser].
Speaking from experience within the mid-level IT projects (not LMP, but I did study IT/LMP and others).
CIO/CTO the more highly certified (and less experience qualified) decisions depend on external corporate-marketeers effectiveness in meetings. The internal/core technology subject mater experts (SME) are considered incompetent/intransigent. The CIO/CTO due to a lack of experience and/or over-inflated career-manager-egos seek the like-minded/capable "YES!" colleagues that support the career-management party line and corporate-marketeer SMEs. Army/LMP, FBI/ISEv5 ... are obviously just a few examples for US, EU may be little better (can't be sure). GIG/NetCentric, from what I here, look like a potentially equal solution of the best that corporate-marketeer SMEs can provide US.
In articles/papers/conversation ... personal studies, CIO/CTO career-managers have been asked by their internal/core technology SMEs to seek Stanford, MIT, GTRI, CMU ... academia help in selecting architectures and products, but ... the corporate-marketeer SMEs are free with advice, easy with promises, and blamestorm insensitive. CIO/CTO career-managers are highly certified and very expensive for US.
I have see some attempts to address the science and technology problems with more titles (venture project manager ...) and certifications (information manager ...). In the short-run it may help, but the highly intelligent career-managers (in a mater of time) catch-on/up with the certification requirements, then the upper-management will promote on certifications and BM/BS/BA... degrees and ignore the successes of worker-bees and pack-mules with good/great performance and experience metrics. The personality-centric qualified (pre-C*O) career-managers in stocks, banking, business, government ... will always game their way to the top of pay/privilege and amazing failures. Hence, IMO, GIG infrastructure and NetCentric interoperability could take decades longer than planned/expected. USA ISE, GIG infrastructure, and NetCentric interoperability are fantastic and needed by US and DOD today, and the technology is available now, but the org/ops structure has (IMO) a potentially crippling steep learning curve with cyberwar enemies starting to devour our ....
SideNote: I still know of CIO/CTO that are confused with what virtualization (infrastructure) and Cloud (services) are going to provide for the technology architecture mixes. I know one decision maker for muti-million$ that knows L/FOSS, freeware, and shareware are all the same kind of products. One industry colleague told me that a decision maker in a recent meeting said DODAF is not used and will never happen, same meeting, DODIEA means nothing within DA and USAF.
I've seen briefing charts where in two bullets succinctly defined the ISE technology-nexus:
Legacy: Technology has defined culture, education, and how information could be applied.
Today: Information will define culture, education, and how technology will be applied.
The above is a significant change equivalent to the move from a blacksmith forge too an industrial foundry, but highly certified CIO/CTO will never notice until collapse/catastrophe. Law, regulations, policies can and should define architectures, but only the worker-bees and pack-mules can build the structures and products of the future. So. $8B loss in a decade (only IMO) is just chump-change [ehhh-haaaa CHA-CHING!]
Requirements creep
Oversight changes induce change/loss of focus
Congress quibbling over federal budget
All-at-once upgrade mentality vs. a continuous flux computer market
Mix in a horribly political workplace environment, at all levels
Microsoft licensing fees thru how many OS "upgrades" & Visual Studio "improvements"
No doubt an incomplete list...
Is this a good recipe for running an efficient process????
SAP was around 18 years ago, but not well-known. And plus, being a German company, it probably would not have been the choice then for a US Defense Department project.
Well, so much for that. From the Navy's "About Navy ERP" page:
The Navy ERP Program uses a product from SAP Corporation, which allows the Navy to unify, standardize, and streamline all its business activities into one completely integrated system.
Is the fact that they are trying to customize off the shelf software. They'd be better off going all FOSS on it, rolling it from the ground up to be honest.
But the other thing - it's all the big boys doing the work, IBM, CA, etc. So they have NO motivation to do it right. Instead they are milking the system for all it's worth.
In a lot of ways we have stacked the deck against the government doing successful projects.
The public gets upset when government workers are highly paid or get nice perks (see the recent GSA Las Vegas conference fiasco), so it is not easy for them to attract top talent.
The public wants more oversight of government projects than is typical for industry. This results in non-standard accounting practices that require custom management software. The accounting requirements can also significantly distort engineering efforts - it is difficult to pay for R&D the benefits multiple programs due to the difficult of knowing what account to charge.
Project cycles are so long that top managers have every incentive to underestimate budgets. If there is a project up for grabs, it goes to the organization that *claims* it can do it most cheaply. If 10 years down the road the project is over budget, the original manager is likely long gone to another project, and probably doesn't even receive any blame for the overruns - that goes to the poor SOB who took over the project without realizing what shape it was actually in.
Government funding is driven by the winds of politics. Projects are created, then delayed, then restarted, then scope changes, the canceled - then revived. Engineering teams are formed and disbanded, then need to be re-formed.
In the end I agree that government projects are inefficient, but I think a lot of the inefficiency comes from misguided attempts by the public to force more efficiency. I don't believe that regulations can make poor engineers into good engineers. If you want good engineers you need to make your jobs more attractive than the jobs offered by the competition.
No surprise here but isn't it funny how the emphasis is always on defense spending? How about the billions of dollars lost to fraud, waste and abuse in government social programs (food stamps, medicare, social security et. al.)? Discretionary defense spending amounts to about $800B/year but social programs are entitlements that add up to $1.3T of the annual budget. Why aren't we pissed about that?
I love examples like this, in a perverse sort of way, because it give me yet another example to give to my students. Government contracts are essentially always done with the "waterfall" model, because the government insists on complete specs before funding. Any competent software engineer educated in the past 20 years knows that waterfall-projects fail if they are above a certain size. Hence, there is an endless supply of examples like this, and will be as long as the government software contracts are managed by people who know nothing about software.
Of course, you can also add in all of the above comments about revolving-door, management turnover, and simple corruption. But even without these, the projects would have failed.
Enjoy life! This is not a dress rehearsal.
reminds me of SpaceX Dragon as compared to LM Orion (ok, bad comparison in many ways), where Dragon was developed along with a launch vehicle and flown twice for about $1 billion (or of that magnitude). Orion is costing billions and billions... not sure when it will ever be flown. What is this difference? An effective cost control is when the money comes out of your own pocket. Unlike the other which is not.
mfwright@batnet.com
Pretty much everything the Pentagon does is over budget, behind schedule, and budget-wise, generally a spawn of wishful thinking
And by "wishful thinking" you mean the wish that a whole bunch of pork will land in a barrel somewhere, right?
Wish? More like planned that way. The Pentagon knows how to play the game: lowball your estimate for a weapon system you're selling as critical to national security, get the process flowing to as many Congressional districts as possible (one factor that raises costs, in fact) in order to gather maximum support, and then when production actually starts, you know that Congress won't have the courage to cancel the program.
I'm very hawkish, but over the years, I've also become very, very cynical about how we buy weapons. This is one of the reasons that, despite my support for free trade in civilian goods, I think perhaps we should go back to a mostly-nationalized weapons building regime. The Navy owns a lot of shipyards, the Air Force a lot of aircraft plants, and the army some armories (and in the past, even armor factories). But they no longer design and build ships, planes, and guns on their own. It's totally contractor driven now, and anyone that studies the issue objectively has to admit that weapons procurement (domestically) is in no way any kind of free market... it just has the appearance of one. The whole process is very corrupt (by design). Maybe we'd be better off going back to designing and building our own ships and aircraft (the Navy especially was into doing this... they even had their own aircraft factory, and they found that it kept costs down in the 20's and 30's as it kept 3rd party contractors honest).
This is coming from a right winger, folks. Entitlements are our biggest budget problem, and a corrupting influence on it's own, but we can not continue to ignore the fiasco that is our arms procurement process and military budget either. No nation in the world can afford $15 billion dollar aircraft carriers and $200+ million dollar fighter planes in any useful quantity. And not only are we engaging in corruption, we're borrowing 40 cents on the dollar to do it.
Life is hard, and the world is cruel
You described a situation where you have people who understand the customer identifying requirements and planning. Makes a lot of sense since an outsider would take many years to understand customer needs. I think the real part is the second part of your statement "ex-military are extremely unlikely to buck the system and stand up to uniformed types" - the problem has nothing to do with uniform, prior service are LESS likely to be impressed with that, what you really have is the age-old problem of not wanting to say no to the customer.
"God fights on the side with the best artillery." - Napoleon, Marshal of France - speaking truth to power
ERP Systems are not that easy to integrate, plus its the military.
I never served in American Armed Forces but I served in another and was the temporary storeman for my platoon for a short while.
It will be an absolutely worst Project Manager's Nightmare Scenario.
Consider a simple rifle , break it down into simple parts
1. Barrel
2. Grip
3. Bolt
4. Rifle Guard
5. Sight Tips
6. Sling
7. Housing
8. Spings & Clips
Then you get accessories
1. Pouch
2. Carry Case
3. Zoom
4. Night Vision
6. Spare Barrels
Dont forget each manufacturer gets its own ID, and each revision of the part gets another, not to mention each batch will be different.
And for most armies, there will be more than one supplier for one part. And talk about colors...
Thats for a rifle, imagine the whole infantry men, from spare weapons, boots, helmets , camouflage, even general issue things like towels and soap
And imagine the number of parts for vehicles. And after youve done all that, don't forget that each unit probably does thing slightly differently from the next unit.
If done on a paper and within the confines of one camp, the complexity is easily managed, but they are doing for the whole DOD. The US armed forces also keep evolving, so you get plenty of new things every few months. So you get new definitions and new types. And there WILL be disagreement on classification.
After that youve got approval, you gotta find the right officer and hope he aint so jumpy to check through the entire list and that he is not too far removed from the specs and men to know what exactly is happening. These guys also move around on their tour of duties, so a new guy comes in every 2-3 years, you gotta give him time to settle and figure out stuff and hope he/she doesnt make changes and throw everything into disarray.
Theres also security, permissions and visibility, and officiers always always delegate such crap. So you gotta figure out a way for their PA(aides). And workflows, what should go into testing, what are the tests? What are the benchmarks and approval process and how each vendor actually supplies. Do you consider delivery and handoff at the base, outpost, unit, or distribution to the men. So imagine the arguments on when is enough, try getting a supply count from the marines who are being shot at.
And dealing with IT inquiries are going to pretty far down the priority list, so whatever cajoling tactic you use in the civilian world , you can chuck it out the window.
Imagine talking to the colonel to get the staff to test and vet the system while they are trying to deal with day to day issues which quite frankly are life and death.
Ive seen commercial erp systems overrun by seven years, and the inventory list is probably less than 0.00001% of what the Dod have to deal with.
Doubt these guys knew what hit them after the contract was signed.
I worked on a few projects that were contracted to the government. One example was where we were tasked with prototyping a replacement for a terminal-based system into a web-based one. We have off-the-shelf software that would save tons of time, and we had to fight tooth and nail to use it (for a prototype, mind you). It was basically an uphill battle anytime we wanted to use anything off-the-shelf (that was our own product!) and tons of time was wasted in endless meetings. Lots of infighting among the gov't people, with too many hands in the pot. Lots of tangential issues slowed down real progress, and we were all unhappy about the end result...
It's just like the US economy: Bush spent away our rainy-day margin to kiss up to various constituents (military, elderly, refund checks). Now he's out of office and the mess is somebody else's.
Table-ized A.I.
Maybe we'd be better off going back to designing and building our own ships and aircraft (the Navy especially was into doing this... they even had their own aircraft factory, and they found that it kept costs down in the 20's and 30's as it kept 3rd party contractors honest).
You know, I've had this thought myself. We've reached the point that we're hiring companies to produce weapons for us where we retain full rights to the technology and design, and are thus the ONLY customer.
If we're going to be funding the design and be the only users, might as well do it entirely in-house. Contract in support when necessary, but have our own company doing it.
I don't read AC A human right
I agree, far too many USA .Gov, .Mil, .Com CIO/CTO are highly certified, and unqualified
Unaccountable leaders are masters, and unrepresented people are slaves. How do US and EU fare?
if you specify the actual cost during the planning phase, then they wouldn't be started in the first place. So people make best case estimates and then reality strikes, the actual cost exceed the allocated budget.
I work in this world and this is absolutely true. Cost estimates and timelines are ridiculously blue-sky, but once you get one started, they're almost impossible to stop. They don't want to have to explain the money spent with zero to show for it. I happen to be on one of these right now. The schedule has slipped 11 months and will probably slip more. Mainly due to the following problem.
Another issue they have is trying to use commercial software (COTS). Beyond the typical office and other desktop applications, It just doesn't work. Commercial companies don't have all the rules, regulations, and odd ball business practices that encumber the government. The long history of failures doesn't stop the (cough) "leadership" (cough) from getting sucked into the fairytale though. They always end up with one of two outcomes: stuck with a customized product with hugely expensive support costs or stuck with a product that doesn't actually do what they need it to do. They should realize they need custom software and go forward with it. I know it's wishful thinking though. I've watched this pendulum go through several cycles. While the folks that got stung by a COTS attempt are in charge, GOTS projects are all you see. Once they move on the new crowd gets sucked in by the slick sales people and the cycle begins again.
I have worked in the belly of this very beast ... and although much wisdom has been shared in this thread, the true solution can be found right here:
http://programming-motherfucker.com/
See you space cowboy
Since costs are often 3-5 times more than planned, it would make since to start off with double the cost, split between 2 competing companies. The one that delivers the best solution gets more contracts in the future. In the long run, this would probably save money.