Slashdot Mirror
The Pentagon's Seven Million Lines of Cobol
MrMetlHed writes "A portion of this Reuters article about the Pentagon's inability to manage paying soldiers properly mentions that their payroll program has 'seven million lines of Cobol code that hasn't been updated.' It goes on to mention that the documentation has been lost, and no one really knows how to update it well. In trying to replace the program, the Pentagon spent a billion dollars and wasn't successful."
But - but - cobol is supposed to be self-documenting!
Army in the 90's
Everyone's pay gets screwed up at least once, then. Uncle Sam takes it back
Mostly minor things like an allowance like jump pay being paid while not on jump status
They had way more soldiers back then today, and payroll did not seem to be a problem. Maybe the Pentagon should go back to using whatever system they had back then.
I've seen this happen with lots of "let's replace this antiquated software" projects. There's alot of trust put into hiring someone to do this properly. Usually the people writing the check don't know enough about software architecture or requirements gathering to foresee that the contractor is going about it the wrong way and dooming the project to failure. Or administration that isn't open to the concepts that must be embraced to move from paperless to electronic, etc. So many ways for something like this to fail terribly. Only time I've seen it succeed is a combination of competent leadership of the software development combined with the administration trusting the judgement of the software developer when it advises on process changes that will need to be implemented. Rarely do you get both of these things together, and when you are missing one then it's a disaster.
I tried to read the article, but it was written in English - a decades-old language.
The code base is so large that its ~4.8 lines per active duty US military person. The code would be shorter if it was nothing but one line per person that prints how much to pay. You might argue that this would have maintenance issues, but automated porting of it would be trivial, and for the $billion they spent try to replace it, you could pay almost $1000 per print statement to keep it updated.
defense spending as a portion of GDP has fallen from about 38% of GDP in 1945 to about 4-5% today
Comparing current spending to WWII spending is either disingenuous or just plain silly.
Added to this, people often have the misconception that just because something is "old" it is less complex than the current requirements. In reality, all that COBOL was written to perform the same function on severely limited hardware that they now want to accomplish on a simple server system -- and I bet the data and processing requirements both then and now are astronomical. The end result is that whoever is doing the new system is likely pitting themselves against whatever the brightest minds of yesteryear were able to produce, and it won't be simple. That old system had time to be fine tuned, and the protocol built up over the years is designed around the precise quirks created by the system. Thus, the entire architecture and ALL related protocol has to be re-examined prior to architecting a replacement system -- and I doubt the winning bidder was even asked to bid on that, especially in a military organization.
If we didn't interfere in other countries' business, so many people around the world wouldn't hate us and we would have no use for such a large military.
"It is a denial of justice not to stretch out a helping hand to the fallen; that is the common right of humanity."
I'm not sure where you got your numbers from, but you may have a couple of items swapped.
In 2012, the US federal government spent $3.56 trillion dollars, with revenues of $2.44 trillion, and a deficit of $1.12 trillion.
Entitlement spending was 61.9%, and defense spending was 18.7% (~ $677 billion).
You can find that data here: Federal Spending by the Numbers - 2012
You can see the long term trend of defense versus entitlement spending here.
much of left-wing thought is a kind of playing with fire by people who don't even know that fire is hot - George Orwell
I've maintained legacy payroll software (Oracle RPT, predates PL/SQL) and have been marginally involved in migrating clients to the new shiny payroll system. Generally it fails where the client wants the new system to behave exactly like the old system.
The new system usually can handle the required business rules (or it's not too much work to make this happen) but all the processes around those rules are different. eg the new system needs to generate report RW200 to lineprinter 6, daily at 6PM and must be formatted just so (no one reads the first 1000 pages, but the summary page is critical to some obscure business process.)
So, the new system has to print unformatted ASCII to a serial line printer, in an obscure way, on nonstandard paper, that's hard to replicate in a modern report writer. Never mind the already written, laser printed, on-demand reports (or emailed, or exported to excel or whatever) have the same information - it's NOT THE SAME - our users will be confused so it MUST BE CHANGED!.
Rinse and repeat for basically everything else in your system and you've heavily modified your new system to behave just like your old payroll system (and killed any performance improvements, worked out all the bugs etc again). because it's so heavily modified you're basically on a unique version of the new system that only certain programmers really understand. Ant they're going to retire / leave because the project was so shit to work on.
Add the usual government oversight/waste and you've blown a billion dollars. (that's impressive though, I have to say.)
This is less of a "government efficiency" issue than it is a "contracting" issue.
Imagine you have a gigantic system like this that you need to replace. So you want to hire someone to build the replacement. You can't just go give $1B to company X and say "I'll see you in five years when you've built me a new pay system" - the taxpayers (and their representatives) would never go for that. Instead, you first go build a set of requirements that such a system must meet and then you award the contract to build the system to the company that convinces you that (a) they'll build the system that best meets those requirements and (b) they'll do so in a cost-competitive way (if not cheapest, then close to cheapest).
Therein lies the crux of the problem - building large complex software systems over multiple years "to spec". In short, it can't really be done:
I've only ever seen one model work successfully (in my time in the USAF and as a contractor):
The above system works beautifully. And Congress, contractors, and contracting agencies within the military hate it. Which is probably why it isn't done more often.
You are correct. And wrong. According to the Reuters article, there were more than 15,000 requirements changes during the lifetime of the project. So you had precisely the right idea. You just underestimated the ability of a bureaucracy to fight back. By an order of magnitude.
And that's what it was, too. Make no mistake, the project failed because a successful software system would reduce the headcount of the DFAS dramatically. That couldn't be allowed to happen, so it was sabotaged by eternal feature creep.
And of course, they started with PeopleSoft, and there's no organization better at absorbing all available money for no return besides the DOD itself. Talk about a match made in hell...
Yes, there was and there is. It's called "source code."
While it's true that COBOL is meant to be self documenting, there is, in a 7 million line project, a difference between understanding any particular section of code and being able to comprehend the entire structure of the project. If that structure is undocumented, you will have a lot of reading before you grasp the program globally. Apparently the "failures" that were being experienced were not leading the maintainers to the appropriate sections of code and such a global understanding had become necessary.
I know it breaks one of the cardinal rules of software development, but if you have a cool billion to throw at the problem and the existing mess cannot be fathomed, perhaps starting afresh is an idea ...
Better to be despised for too anxious apprehensions, than ruined by too confident a security. --Edmund Burke