Slashdot Mirror

← Back to Stories (view on slashdot.org)

The Pentagon's Seven Million Lines of Cobol

Posted by Soulskill on from the lords-of-cobol-hear-my-prayers dept.

MrMetlHed writes "A portion of this Reuters article about the Pentagon's inability to manage paying soldiers properly mentions that their payroll program has 'seven million lines of Cobol code that hasn't been updated.' It goes on to mention that the documentation has been lost, and no one really knows how to update it well. In trying to replace the program, the Pentagon spent a billion dollars and wasn't successful."

21 of 345 comments (clear)

  1. Corruption by Anonymous Coward · · Score: 3, Insightful

    the Pentagon spent a billion dollars and wasn't successful

    Sounds like corruption. If you can't wrangle up a team of coders and project managers with a billion dollar carrot, then there is something terribly internally wrong with your process.

  2. How did the military pay during WW2? by Spy+Handler · · Score: 5, Insightful

    They had way more soldiers back then today, and payroll did not seem to be a problem. Maybe the Pentagon should go back to using whatever system they had back then.

    1. Re: How did the military pay during WW2? by jfdavis668 · · Score: 2, Insightful

      The used 100 to 1000 times as many people. Working in un-air conditioned building doing routine work over and over. Then it was all double and triple checked. When they screwed up, you could yell at them. There is no way we could afford to recreate that system with today's pay scales. Also, these were sharp people. No one like that would even apply for the job. Welcome to the post industrial society.

  3. Re:Typical government efficiency... by Anonymous Coward · · Score: 4, Insightful

    I've seen this happen with lots of "let's replace this antiquated software" projects. There's alot of trust put into hiring someone to do this properly. Usually the people writing the check don't know enough about software architecture or requirements gathering to foresee that the contractor is going about it the wrong way and dooming the project to failure. Or administration that isn't open to the concepts that must be embraced to move from paperless to electronic, etc. So many ways for something like this to fail terribly. Only time I've seen it succeed is a combination of competent leadership of the software development combined with the administration trusting the judgement of the software developer when it advises on process changes that will need to be implemented. Rarely do you get both of these things together, and when you are missing one then it's a disaster.

  4. Re:Cobol is self-documenting by Avidiax · · Score: 5, Insightful

    The claim that the documentation "vanished" seems bogus. Far more likely in my opinion that it never existed in the first place, or that at some point they fired everyone, and thus broke the chain of custody.

  5. Re:Cobol is self-documenting by cold+fjord · · Score: 4, Insightful

    Normal staff turnover and one building move would probably stand a good chance of taking care of it.

    --
    much of left-wing thought is a kind of playing with fire by people who don't even know that fire is hot - George Orwell
  6. Re:Typical government efficiency... by ebno-10db · · Score: 4, Insightful

    defense spending as a portion of GDP has fallen from about 38% of GDP in 1945 to about 4-5% today

    Comparing current spending to WWII spending is either disingenuous or just plain silly.

  7. Cobol really is self-documenting by techno-vampire · · Score: 3, Insightful

    Eh...there probably was some half baked documentation at some point,

    Yes, there was and there is. It's called "source code." One of the reasons that COBOL is such a verbose language is that it was designed so that bean counters with no programming experience could audit the source code and understand it well enough to make sure that nobody was stealing anything. Not only that, it's rare that COBOL code actually needs any comments because the variable names are long enough that you shouldn't ever have to guess what any of them are used for or what's being done with/to them.

    As far as spaghetti code goes, that can be a problem, especially in very old code, from before such things as structured programming were conceived. And, there's even a statement, "ALTER," which allows you to create self-modifying code, although even back in the '80s when I learned it in school, we were warned never to use it.

    --
    Good, inexpensive web hosting
    1. Re:Cobol really is self-documenting by Capsaicin · · Score: 5, Insightful

      Yes, there was and there is. It's called "source code."

      While it's true that COBOL is meant to be self documenting, there is, in a 7 million line project, a difference between understanding any particular section of code and being able to comprehend the entire structure of the project. If that structure is undocumented, you will have a lot of reading before you grasp the program globally. Apparently the "failures" that were being experienced were not leading the maintainers to the appropriate sections of code and such a global understanding had become necessary.

      I know it breaks one of the cardinal rules of software development, but if you have a cool billion to throw at the problem and the existing mess cannot be fathomed, perhaps starting afresh is an idea ...

      --
      Better to be despised for too anxious apprehensions, than ruined by too confident a security. --Edmund Burke
    2. Re:Cobol really is self-documenting by techno-vampire · · Score: 3, Insightful

      COMPUTE t1(i1) = (t2(i2) ** 3) / (t3(i1) * 2 + 1);

      Yes, except that you'd have a period at the end, not a semicolon. (This is why statements in COBOL are referred to as "sentences.") However, if you did try something like that in a shop where the bean counters were auditing your code, they'd probably reject it for lack of clarity and insist that you gave that variable a more meaningful name. If you really want to, you can obfuscate code written in any language, but it's probably harder to get away with in COBOL than in most other languages.

      --
      Good, inexpensive web hosting
  8. Re:Typical government efficiency... by Em+Adespoton · · Score: 5, Insightful

    Added to this, people often have the misconception that just because something is "old" it is less complex than the current requirements. In reality, all that COBOL was written to perform the same function on severely limited hardware that they now want to accomplish on a simple server system -- and I bet the data and processing requirements both then and now are astronomical. The end result is that whoever is doing the new system is likely pitting themselves against whatever the brightest minds of yesteryear were able to produce, and it won't be simple. That old system had time to be fine tuned, and the protocol built up over the years is designed around the precise quirks created by the system. Thus, the entire architecture and ALL related protocol has to be re-examined prior to architecting a replacement system -- and I doubt the winning bidder was even asked to bid on that, especially in a military organization.

  9. Re:Typical government efficiency... by pwizard2 · · Score: 4, Insightful

    If we didn't interfere in other countries' business, so many people around the world wouldn't hate us and we would have no use for such a large military.

    --
    "It is a denial of justice not to stretch out a helping hand to the fallen; that is the common right of humanity."
  10. Re:Typical government efficiency... by dan_barrett · · Score: 4, Insightful

    I've maintained legacy payroll software (Oracle RPT, predates PL/SQL) and have been marginally involved in migrating clients to the new shiny payroll system. Generally it fails where the client wants the new system to behave exactly like the old system.

    The new system usually can handle the required business rules (or it's not too much work to make this happen) but all the processes around those rules are different. eg the new system needs to generate report RW200 to lineprinter 6, daily at 6PM and must be formatted just so (no one reads the first 1000 pages, but the summary page is critical to some obscure business process.)

    So, the new system has to print unformatted ASCII to a serial line printer, in an obscure way, on nonstandard paper, that's hard to replicate in a modern report writer. Never mind the already written, laser printed, on-demand reports (or emailed, or exported to excel or whatever) have the same information - it's NOT THE SAME - our users will be confused so it MUST BE CHANGED!.

    Rinse and repeat for basically everything else in your system and you've heavily modified your new system to behave just like your old payroll system (and killed any performance improvements, worked out all the bugs etc again). because it's so heavily modified you're basically on a unique version of the new system that only certain programmers really understand. Ant they're going to retire / leave because the project was so shit to work on.

    Add the usual government oversight/waste and you've blown a billion dollars. (that's impressive though, I have to say.)

  11. Re:Typical government efficiency... by AndrewLarge · · Score: 5, Insightful

    This is less of a "government efficiency" issue than it is a "contracting" issue.

    Imagine you have a gigantic system like this that you need to replace. So you want to hire someone to build the replacement. You can't just go give $1B to company X and say "I'll see you in five years when you've built me a new pay system" - the taxpayers (and their representatives) would never go for that. Instead, you first go build a set of requirements that such a system must meet and then you award the contract to build the system to the company that convinces you that (a) they'll build the system that best meets those requirements and (b) they'll do so in a cost-competitive way (if not cheapest, then close to cheapest).

    Therein lies the crux of the problem - building large complex software systems over multiple years "to spec". In short, it can't really be done:

    • 1. You won't get the requirements right. At best, if you spend a ton of money, you'll get a decent set of requirements, for some narrow segment of your user base, that is appropriate for the state of the world at contract award. But they'll be woefully out of date by the time the system is actually built. More often, you won't pay enough money and you'll get an RFP that is 50%+ "cut and paste" from previous RFPs, and has only a passing resemblance to what is really needed.
    • 2. Managing a multi-year software development project is tough enough when you're the one building it. It is many times more difficult to try and look over a contractor's shoulder and make sure that they are (a) meeting requirements, (b) meeting schedule, and (c) not wasting (or stealing) money. This is even more difficult when you have thousands of crappy nonsensical or ambiguous requirements (the norm for such large systems).
    • 3. Even if you get decent requirements and decent/lucky contract management, you still have huge product and engineering hurdles that don't often show themselves until you get the software in the hands of real users. And (more often as not) find out that it doesn't work for them.

    I've only ever seen one model work successfully (in my time in the USAF and as a contractor):

    • 1. Put organic (e.g., military or civil service, not contractor) resources in charge of the system development.
    • 2. Don't try to spec and build the system as a whole unit. Instead, start with something small and easily defined and then work *directly* with end users to evolve and enhance the system over multiple years. At some point in the evolution of the system (neither the beginning nor the end), you make the call that the system is functional and stable enough to deploy.
    • 3. Use contractors as labor, project-managed by your organic resources. This tends to mean fairly integrated teams (organic and contractor) and a different sort of contract vehicle. It also means that swapping contractors out is actually feasible and doesn't cause the failure of the entire project.

    The above system works beautifully. And Congress, contractors, and contracting agencies within the military hate it. Which is probably why it isn't done more often.

  12. Re:Cobol is self-documenting by budgenator · · Score: 4, Insightful

    Well the source code is usually fairly legible, but at 7 million lines the spaghetti factor is likely pretty high.

    --
    Apocalypse Cancelled, Sorry, No Ticket Refunds
  13. Re:Cobol is self-documenting by Ashenkase · · Score: 3, Insightful

    In trying to replace the program, the Pentagon spent a billion dollars and wasn't successful.

    Translation: the pimps deployed highly effective counter measures to shock and awe the client, the result was a resounding victory of "extended" contracts!

  14. Re:Cobol is self-documenting by Chris+Mattern · · Score: 3, Insightful

    but it has the advantage of being quite easily maintained.

    Ahhhhh...no. COBOL has a GOTO. Legacy COBOL has a tendency to use the GOTO a lot. In legacy COBOL, all the variables are global (although you did at least have to declare them all). These things do not make for easily maintained code.

  15. Re:Cobol is self-documenting by __aaltlg1547 · · Score: 4, Insightful

    Normal staff turnover in the military is 2 years on an assignment. People come in knowing next to nothing, spend two years learning 80% of what the previous incument knew and then move on. It doesn't take many cycles of that before all useful knowledge is lost.

  16. Re:Typical government efficiency... by jbohumil · · Score: 3, Insightful

    You totally get it. It takes strong leadership to tell the business units that they must conform their procedures to the new system and it will be installed as is. That is the way to succeed. Then once it's in you can start opening it up.

  17. Re:Cobol is self-documenting by S1ngularity · · Score: 3, Insightful

    exactly how much of it is actually active?

    For every line of active duty Cobol code, there are seven support lines behind the front.

  18. Re:Cobol is self-documenting by kmoser · · Score: 3, Insightful

    Well the source code is usually fairly legible, but at 7 million lines the spaghetti factor is likely pretty high.

    You assume the source code was still available.