Slashdot Mirror

← Back to Stories (view on slashdot.org)

The Pentagon's Seven Million Lines of Cobol

Posted by Soulskill on from the lords-of-cobol-hear-my-prayers dept.

MrMetlHed writes "A portion of this Reuters article about the Pentagon's inability to manage paying soldiers properly mentions that their payroll program has 'seven million lines of Cobol code that hasn't been updated.' It goes on to mention that the documentation has been lost, and no one really knows how to update it well. In trying to replace the program, the Pentagon spent a billion dollars and wasn't successful."

73 of 345 comments (clear)

  1. Cobol is self-documenting by mveloso · · Score: 5, Funny

    But - but - cobol is supposed to be self-documenting!

    1. Re:Cobol is self-documenting by Anonymous Coward · · Score: 5, Funny

      But - but - cobol is supposed to be self-documenting!

      TL;DR

    2. Re:Cobol is self-documenting by Avidiax · · Score: 5, Insightful

      The claim that the documentation "vanished" seems bogus. Far more likely in my opinion that it never existed in the first place, or that at some point they fired everyone, and thus broke the chain of custody.

    3. Re:Cobol is self-documenting by cold+fjord · · Score: 2

      Maybe at the level of modules or minor subsystems, but with software systems that large you would still want architecture documents and data definitions as a minimum.

      --
      much of left-wing thought is a kind of playing with fire by people who don't even know that fire is hot - George Orwell
    4. Re:Cobol is self-documenting by cold+fjord · · Score: 4, Insightful

      Normal staff turnover and one building move would probably stand a good chance of taking care of it.

      --
      much of left-wing thought is a kind of playing with fire by people who don't even know that fire is hot - George Orwell
    5. Re:Cobol is self-documenting by TheCaptain · · Score: 4, Interesting

      Eh...there probably was some half baked documentation at some point, but I doubt it was maintained very well by the people who edited that codebase over the decades.

      I also doubt they fired any of them unless they were contractors...you have no idea how ugly the federal workers union is about things like this. They almost can't lose their jobs through incompetence or anything else. Which brings me to the problem...the people who wrote it probably wrote half-assed spaghetti code, didn't document it well, and then died off or retired. No one is learning cobol anymore, so you get what we've got right here.

      Plus...trying to replace any system in the government or military is an extremely painful exercise that probably fails more often than it succeeds. Between the people you need to deal with, and the policies you need to dance around...it's almost impossible to stand a new system up. (Unless you have someone in a high place that really gets it, and champions the hell out of it...and even then, it's iffy.)

      I was a defense contractor for 4 or 5 years. It was quite a few years ago now. Left that behind, and I don't miss it.

    6. Re:Cobol is self-documenting by interval1066 · · Score: 3, Informative

      Far more likely in my opinion that it never existed in the first place, or that at some point they fired everyone, and thus broke the chain of custody.

      Being the spouse of some one who works for a Gov. entity (her) AND being in IT (me), its far more likely that the the engineers who created the system(s) have long since retired. Fed workers rarely get 'fired'. And interestingly its more likely that the system was documented to a much higher degree than you would think; there are entire fed. departments devoted to documenting things and creating requirements documents. The problem is once the process is documented and archived, those same sad COBOL systems are used to process the records that describe the location of the documentation

      --
      Python: 'And then suddenly you have a language which says "we're all stuck with whatever the whiniest coder wants".'
    7. Re:Cobol is self-documenting by plover · · Score: 3, Funny

      The claim that the documentation "vanished" seems bogus. Far more likely in my opinion that it never existed in the first place, or that at some point they fired everyone, and thus broke the chain of custody.

      I think the truth is probably much simpler than that. Someone dropped the card deck containing the documentation, and they never managed to sort it back into the right order.

      --
      John
    8. Re:Cobol is self-documenting by Jane+Q.+Public · · Score: 2

      "The claim that the documentation "vanished" seems bogus. Far more likely in my opinion that it never existed in the first place, or that at some point they fired everyone, and thus broke the chain of custody."

      I agree with the latter, but that still means the documentation did "vanish". It just doesn't address why.

      I think the money would have been better spent simply replacing the Pentagon and all its staff. The new one can be 5-sided, too. But they should build it in Montana and staff it with locals. It's cheaper. They'd probably win more wars for us too.

    9. Re:Cobol is self-documenting by BrokenHalo · · Score: 2

      Eh...there probably was some half baked documentation at some point, but I doubt it was maintained very well

      That may well be because any decent COBOL programmer knows (or knew) that the code itself is indeed self-documenting. COBOL code may be tedious to write (actually, it certainly is) but it has the advantage of being quite easily maintained.

      So, no excuses. It's more likely the US Govt is too cheap to hire anyone with the appropriate skills. It's also quite possible that an age bias has been applied by their HR bozos, thus excluding that generation most likely to be good at this work.

    10. Re:COBOL is self-documenting by ebno-10db · · Score: 5, Funny

      Yes, but every line is documented the same way, "don't touch this because I don't know what it does but sometimes it works".

    11. Re:Cobol is self-documenting by budgenator · · Score: 4, Insightful

      Well the source code is usually fairly legible, but at 7 million lines the spaghetti factor is likely pretty high.

      --
      Apocalypse Cancelled, Sorry, No Ticket Refunds
    12. Re:Cobol is self-documenting by Ashenkase · · Score: 3, Insightful

      In trying to replace the program, the Pentagon spent a billion dollars and wasn't successful.

      Translation: the pimps deployed highly effective counter measures to shock and awe the client, the result was a resounding victory of "extended" contracts!

    13. Re:Cobol is self-documenting by mrmeval · · Score: 2

      Hughes would compartmentalize documentation on how to manufacture everything. There software had similar fragmentation in the source code. To collect all the documents would require having access to the root document which would then lead to the first level. Then would come tertiary and quaternary levels. There is NO centralized list of all the documents needed, you MUST parse every sheet of paper.

      I recall one gasket for a keyboard for some aircraft system they'd installed in the mesozoic era. When Raytheon absorbed and gagged on them they'd done an incredible job of digitizing and indexing all of that documentation. Even with that Herculean effort the drawing on how to make the gasket which had dimensions and the material was completely lost. The government did not afai could find out had never gotten complete plans for that system. They had to recreate that piece of paper at a cost of 55,000 dollars.

      Nah, don't think Raytheon is all puppies and kittens, they're just more efficient at sucking money out of government.

      --
      I'd go on a Vegan diet but the delivery time from Vega is too long. --brownkitty
    14. Re:Cobol is self-documenting by Anonymous Coward · · Score: 5, Funny

      Seven million lines of COBOL might only be a short sorting routine.

    15. Re:Cobol is self-documenting by Chris+Mattern · · Score: 3, Insightful

      but it has the advantage of being quite easily maintained.

      Ahhhhh...no. COBOL has a GOTO. Legacy COBOL has a tendency to use the GOTO a lot. In legacy COBOL, all the variables are global (although you did at least have to declare them all). These things do not make for easily maintained code.

    16. Re:Cobol is self-documenting by budgenator · · Score: 3, Interesting

      We did flowcharts in Basic and Fortran, but COBOL was IPO charts (Input, Process, Output) and module diagrams for us; oh we did psudeocode of the modules, psuedocoding COBOL is really tough because cobol is so wordy.

      --
      Apocalypse Cancelled, Sorry, No Ticket Refunds
    17. Re:Cobol is self-documenting by __aaltlg1547 · · Score: 4, Insightful

      Normal staff turnover in the military is 2 years on an assignment. People come in knowing next to nothing, spend two years learning 80% of what the previous incument knew and then move on. It doesn't take many cycles of that before all useful knowledge is lost.

    18. Re:Cobol is self-documenting by plopez · · Score: 2

      " As for the 7 million lines of code, exactly how much of it is actually active?"

      That's the multi-billion dollar question, now isn't it?

      --
      putting the 'B' in LGBTQ+
    19. Re:Cobol is self-documenting by jbohumil · · Score: 2

      I work on Cobol systems where the documentation has literally yellowed and begun to disintegrate.

    20. Re:Cobol is self-documenting by RoknrolZombie · · Score: 2

      I will say that I was pleased to discover that my old software (my very first program, written with one of my NCO's - and it's not clear from the comment, but that wasn't anywhere close to my job...it just happened for convenience) was still being used more than 10 years after the fact, although I do admit that I was a bit dismayed as well...in retrospect it wasn't all that impressive :p

    21. Re: Cobol is self-documenting by Mabhatter · · Score: 3, Funny

      Those tapes got wiped when the magnetic alien was stored too close.

    22. Re:Cobol is self-documenting by Anne+Thwacks · · Score: 3, Funny

      Have you seen Cobol? It takes several hundred lines to write a "Hello World" program. 7 million lines of Cobol can probably be replaced by 70 lines of Perl (at the expense of any possibility of anyone ever reading it).

      --
      Sent from my ASR33 using ASCII
    23. Re:Cobol is self-documenting by neonsignal · · Score: 2

      I'm no fan of COBOL (and enjoyed Dijkstra's criticism), but its main flaws are not to do with being verbose. "Hello World" is longer than it would be in most languages, though a lot of that is just overhead; in general the line count in COBOL is not massively bigger than in other languages. Of course, the syntax in each line can be a mouthful, with an excessive use of keywords obscuring the structure, but in coding business logic this is not always a disadvantage (business logic is often convoluted anyway, and at least the code fragments are a little more readable to a layperson).

      My point is, although you are probably correct that there is very likely a better choice of a modern language to replace this code, it is going to be a lot more than 70 lines worth. The real saving in code is most likely to come through the refactoring process, whatever language is chosen.

    24. Re:Cobol is self-documenting by S1ngularity · · Score: 3, Insightful

      exactly how much of it is actually active?

      For every line of active duty Cobol code, there are seven support lines behind the front.

    25. Re:Cobol is self-documenting by kmoser · · Score: 3, Insightful

      Well the source code is usually fairly legible, but at 7 million lines the spaghetti factor is likely pretty high.

      You assume the source code was still available.

    26. Re:Cobol is self-documenting by budgenator · · Score: 2

      Have you seen Cobol? It takes several hundred lines to write a "Hello World" program. 7 million lines of Cobol can probably be replaced by 70 lines of Perl (at the expense of any possibility of anyone ever reading it).

      Hello World isn't so bad

        IDENTIFICATION DIVISION.
                  PROGRAM-ID. HELLO-WORLD.
                  PROCEDURE DIVISION.
                          DISPLAY 'Hello, world'.
                          STOP RUN.

      Even Hello, OS/360 circa 1972 is

          001 IDENTIFICATION DIVISION.
          002 PROGRAM-ID. 'HELLO'.
          003 ENVIRONMENT DIVISION.
          004 CONFIGURATION SECTION.
          005 SOURCE-COMPUTER. IBM-360.
          006 OBJECT-COMPUTER. IBM-360.
          0065 SPECIAL-NAMES.
          0066 CONSOLE IS CNSL.
          007 DATA DIVISION.
          008 WORKING-STORAGE SECTION.
          009 77 HELLO-CONST PIC X(12) VALUE 'HELLO, WORLD'.
          075 PROCEDURE DIVISION.
          090 000-DISPLAY.
          100 DISPLAY HELLO-CONST UPON CNSL.
          110 STOP RUN.

      --
      Apocalypse Cancelled, Sorry, No Ticket Refunds
  2. COBOL is self-documenting by davidwr · · Score: 2

    At least that's what my old B-school roommate told me.

    --
    Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
  3. Typical government efficiency... by intermodal · · Score: 3, Interesting

    a billion dollars to replace an antiquated program and the project fails. This is why our military is the most expensive in the world, and why I've argued for years that comparing military spending between nations is only apples to apples if each nation is competently spending what they are given.

    --
    In SOVIET RUSSIA... erm...NSA AMERICA, the Internet logs onto YOU!
    1. Re:Typical government efficiency... by Anonymous Coward · · Score: 4, Insightful

      I've seen this happen with lots of "let's replace this antiquated software" projects. There's alot of trust put into hiring someone to do this properly. Usually the people writing the check don't know enough about software architecture or requirements gathering to foresee that the contractor is going about it the wrong way and dooming the project to failure. Or administration that isn't open to the concepts that must be embraced to move from paperless to electronic, etc. So many ways for something like this to fail terribly. Only time I've seen it succeed is a combination of competent leadership of the software development combined with the administration trusting the judgement of the software developer when it advises on process changes that will need to be implemented. Rarely do you get both of these things together, and when you are missing one then it's a disaster.

    2. Re:Typical government efficiency... by ebno-10db · · Score: 4, Insightful

      defense spending as a portion of GDP has fallen from about 38% of GDP in 1945 to about 4-5% today

      Comparing current spending to WWII spending is either disingenuous or just plain silly.

    3. Re:Typical government efficiency... by tnk1 · · Score: 2

      Apples to apples is always a good comparison, but exactly what military are you saying is more efficiently run? The US military may waste more money per year than the GDP of some small countries, but it is probably not the most proportionally wasteful military out there.

      It's also important to note that as the mission of the military becomes more expansive, there is a lot more room for waste. If you're running the military of say, Luxembourg, you might be able to keep a tighter leash on it, not just because it is smaller, but also because I doubt it leaves Luxembourg much, and when it does, it probably doesn't go farther than next-door for training exercises.

      If you're talking about the military that has the most missions of any military on Earth, it's really, really difficult to find someone who has a comparable profile. Only the USSR's armed forces really ever came close, and China isn't there yet. The other NATO countries just don't have the same mission and only the UK and France have any pretensions towards serious power projection.

    4. Re:Typical government efficiency... by Em+Adespoton · · Score: 5, Insightful

      Added to this, people often have the misconception that just because something is "old" it is less complex than the current requirements. In reality, all that COBOL was written to perform the same function on severely limited hardware that they now want to accomplish on a simple server system -- and I bet the data and processing requirements both then and now are astronomical. The end result is that whoever is doing the new system is likely pitting themselves against whatever the brightest minds of yesteryear were able to produce, and it won't be simple. That old system had time to be fine tuned, and the protocol built up over the years is designed around the precise quirks created by the system. Thus, the entire architecture and ALL related protocol has to be re-examined prior to architecting a replacement system -- and I doubt the winning bidder was even asked to bid on that, especially in a military organization.

    5. Re:Typical government efficiency... by pwizard2 · · Score: 4, Insightful

      If we didn't interfere in other countries' business, so many people around the world wouldn't hate us and we would have no use for such a large military.

      --
      "It is a denial of justice not to stretch out a helping hand to the fallen; that is the common right of humanity."
    6. Re:Typical government efficiency... by cold+fjord · · Score: 4, Informative

      I'm not sure where you got your numbers from, but you may have a couple of items swapped.

      In 2012, the US federal government spent $3.56 trillion dollars, with revenues of $2.44 trillion, and a deficit of $1.12 trillion.

      Entitlement spending was 61.9%, and defense spending was 18.7% (~ $677 billion).

      You can find that data here: Federal Spending by the Numbers - 2012

      You can see the long term trend of defense versus entitlement spending here.

      --
      much of left-wing thought is a kind of playing with fire by people who don't even know that fire is hot - George Orwell
    7. Re:Typical government efficiency... by dan_barrett · · Score: 4, Insightful

      I've maintained legacy payroll software (Oracle RPT, predates PL/SQL) and have been marginally involved in migrating clients to the new shiny payroll system. Generally it fails where the client wants the new system to behave exactly like the old system.

      The new system usually can handle the required business rules (or it's not too much work to make this happen) but all the processes around those rules are different. eg the new system needs to generate report RW200 to lineprinter 6, daily at 6PM and must be formatted just so (no one reads the first 1000 pages, but the summary page is critical to some obscure business process.)

      So, the new system has to print unformatted ASCII to a serial line printer, in an obscure way, on nonstandard paper, that's hard to replicate in a modern report writer. Never mind the already written, laser printed, on-demand reports (or emailed, or exported to excel or whatever) have the same information - it's NOT THE SAME - our users will be confused so it MUST BE CHANGED!.

      Rinse and repeat for basically everything else in your system and you've heavily modified your new system to behave just like your old payroll system (and killed any performance improvements, worked out all the bugs etc again). because it's so heavily modified you're basically on a unique version of the new system that only certain programmers really understand. Ant they're going to retire / leave because the project was so shit to work on.

      Add the usual government oversight/waste and you've blown a billion dollars. (that's impressive though, I have to say.)

    8. Re:Typical government efficiency... by cold+fjord · · Score: 2

      The government doesn't create the GDP, but it does tax economic activity reflected in the GDP and various types of wealth. The government uses taxes to get the money it spends. The tax burden imposed by the government on the economy may be represented as a percentage of the GDP. Maybe you've seen some faulty discussions of the issue in the past, but it is a fairly common way of describing the burden of government and taxes on an economy, and on society. That isn't a claim that the government creates the GDP.

      --
      much of left-wing thought is a kind of playing with fire by people who don't even know that fire is hot - George Orwell
    9. Re:Typical government efficiency... by gmueckl · · Score: 2

      This reminds me of the odd undocumented workaround processes that always seem to creep into the work habits of the users of some complex systems. It is almost impossible to catch those when gathering requirements because they never formally exist.

      --
      http://www.moonlight3d.eu/
    10. Re:Typical government efficiency... by AndrewLarge · · Score: 5, Insightful

      This is less of a "government efficiency" issue than it is a "contracting" issue.

      Imagine you have a gigantic system like this that you need to replace. So you want to hire someone to build the replacement. You can't just go give $1B to company X and say "I'll see you in five years when you've built me a new pay system" - the taxpayers (and their representatives) would never go for that. Instead, you first go build a set of requirements that such a system must meet and then you award the contract to build the system to the company that convinces you that (a) they'll build the system that best meets those requirements and (b) they'll do so in a cost-competitive way (if not cheapest, then close to cheapest).

      Therein lies the crux of the problem - building large complex software systems over multiple years "to spec". In short, it can't really be done:

      • 1. You won't get the requirements right. At best, if you spend a ton of money, you'll get a decent set of requirements, for some narrow segment of your user base, that is appropriate for the state of the world at contract award. But they'll be woefully out of date by the time the system is actually built. More often, you won't pay enough money and you'll get an RFP that is 50%+ "cut and paste" from previous RFPs, and has only a passing resemblance to what is really needed.
      • 2. Managing a multi-year software development project is tough enough when you're the one building it. It is many times more difficult to try and look over a contractor's shoulder and make sure that they are (a) meeting requirements, (b) meeting schedule, and (c) not wasting (or stealing) money. This is even more difficult when you have thousands of crappy nonsensical or ambiguous requirements (the norm for such large systems).
      • 3. Even if you get decent requirements and decent/lucky contract management, you still have huge product and engineering hurdles that don't often show themselves until you get the software in the hands of real users. And (more often as not) find out that it doesn't work for them.

      I've only ever seen one model work successfully (in my time in the USAF and as a contractor):

      • 1. Put organic (e.g., military or civil service, not contractor) resources in charge of the system development.
      • 2. Don't try to spec and build the system as a whole unit. Instead, start with something small and easily defined and then work *directly* with end users to evolve and enhance the system over multiple years. At some point in the evolution of the system (neither the beginning nor the end), you make the call that the system is functional and stable enough to deploy.
      • 3. Use contractors as labor, project-managed by your organic resources. This tends to mean fairly integrated teams (organic and contractor) and a different sort of contract vehicle. It also means that swapping contractors out is actually feasible and doesn't cause the failure of the entire project.

      The above system works beautifully. And Congress, contractors, and contracting agencies within the military hate it. Which is probably why it isn't done more often.

    11. Re:Typical government efficiency... by Antique+Geekmeister · · Score: 2

      I've been technical lead and technical contributor on numerous such projects, it's an absolutely core part of my work. And it is incredibly difficult in large environments, where numerous groups have evolved distinct usage and workflows and are often very resistant to change. Coupled with the amount of money being managed in this project, and the military and security requirements, such a project is well beyond the capabilities of any group I've ever met.

      Trust is not a sufficient factor, I'm afraid. Competence, and communications among the groups, is critical. That requires corporate buyin, and ledership and technical and business acumen that are extremely rare and will not be found by a "lowest bid" process.

    12. Re:Typical government efficiency... by budgenator · · Score: 2

      COBOL is a high-level language that had the first compilers in 1960 running essentially the same program on both RCA and UNIVAC computers. Usualy some changes may have to be made in the CONFIGURATION SECTION. but other than that the compiler and the Operating system takes care of it. The problems I'd imagine happening is some of it is written in pre-ANSI COBOL 68, most in ANSI COBOL 68, a fair amount in COBOL 1974 and a smattering of COBOL 1985; so a lot can depend on how strict the compiler is. I imagine every time Congress changes the tax code, the military gets a pay raise or contributory entitlement change; some COBOL programers develope PTSD and the program gets an orphaned module or two. After several decade on maintence and changes it probably a nightmare figuring out what gets used and whats never called.

      --
      Apocalypse Cancelled, Sorry, No Ticket Refunds
    13. Re:Typical government efficiency... by dcollins · · Score: 2

      Comparing the world-beating Nazi military & engineering structure to some guys in caves who arguably aren't even an actual organization is lunacy.

      But fine, let's use your "real war" coffins as a metric. U.S. military deaths in WWII: 416,800. U.S. military deaths in Iraq & Afghanistan combined: 4,409+2,200 = 6,609 (source: Wikipedia). Using the military GDP numbers from upthread, we have GDP-Percent-Per-Coffin values:

      WWII -- 38/416800 = 0.00009
      Post 9/11 -- 4.5/6609 = 0.00068

      So by your "real war" measure, the GDP-Percent-Per-Coffin expenditure has risen over 7-fold since WWII. The post-9/11 military situation is like about 1/60th as significant as WWII by the coffins metric, and your equating of the two is comically outlandish.

      --
      We know where leadership by an anti-intellectual "strongman" who scapegoats minorities and likes boisterous rallies goes
    14. Re:Typical government efficiency... by bored · · Score: 2

      You are quoting the heritage foundation, which has a specific agenda. If you don't know what it is then that is your problem.

      How about I quote the war resisters league, where they claim 47% of the government spending is military related.

      The difference between the two numbers is, what one considers military spending, and what considers government spending in general. For example, is the NSA's gigantic surveillance military spending (it is after all part of the "defense department")? How about the pension program for veterans?

      A rational mind would say they both are part of the military, the second is simply part of the benefits program we give to our soldiers who risk their lives. A built in cost because at some point, someone said it was a damn good idea to take care of soldiers when they come home lest they become the troops for the next revolution.

      So, the truth is probably somewhere in the middle, although aside from the removal of the social security program (which isn't directly part of your income taxes) which I think is a little disingenuous, I think the war resisters league gets it a little more accurate because they count things like the portion of NASA's budget used to launch military devices and similar things.

    15. Re:Typical government efficiency... by cold+fjord · · Score: 2

      The true indecency was the attacks against innocent people by al Qaida - they killed and wounded thousands of people before the US became truly committed to fighting them. There is no shame in calling the war what it is. It isn't a witch hunt. There have been hundreds of arrests and convictions for terrorism related offenses in the US. There have been thousands and thousands of terrorists killed or captured abroad. The reason the day to day threat to American lives appears to be low is due to the effectiveness of the military, intelligence, and police efforts. Don't kid yourself - al Qaida and its affiliates were responsible for killing tens of thousands of people in Iraq, and many more around the world. They would do it in the US if they could figure out how. And do note that there are al Qaida affiliated groups fighting in Syria. Both the rebels and the Syrian government have used chemical weapons. If al Qaida manages to get their hands on those and smuggles them out of Syria, they might very well try to use them in the US. That could easily kill hundreds, or thousands of people in a single attack.

      --
      much of left-wing thought is a kind of playing with fire by people who don't even know that fire is hot - George Orwell
    16. Re:Typical government efficiency... by jbohumil · · Score: 3, Insightful

      You totally get it. It takes strong leadership to tell the business units that they must conform their procedures to the new system and it will be installed as is. That is the way to succeed. Then once it's in you can start opening it up.

    17. Re: Typical government efficiency... by Mabhatter · · Score: 2

      There is no "off the shelf" product that is going to handle payments to Civil War survivors and the 160 years of federal laws passed since then for paying soldiers/survivors their wages in every war since.

    18. Re: Typical government efficiency... by Cramer · · Score: 2

      Bullshit. At the simplest, one need know only who to pay, how much, and how often. Information any accounting package can handle with ease. The problem is not finding a system -- tho they'll happily burn through millions over many years "looking at options"; the problem is getting from the old system to the new system, without making an even bigger mess.

      There are private firms with more complex payroll systems than the US Military. Yet, they aren't running century old software on systems no one understands. It's not cost them billions totaled over the entire life of the company. EDS and ADP run payroll for *THOUSANDS* of companies, and they turn a huge profit on it.

    19. Re: Typical government efficiency... by Hognoxious · · Score: 2

      At the simplest, one need know only who to pay, how much, and how often.

      Well said.

      Likewise, to fly a 747 one need only push and pull the stick a bit, to perform a heart transplant one need only know what to cut and what to stitch, and to cure cancer one need only know what atoms to stick together.

      --
      Confucius say, "Find worm in apple - bad. Find half a worm - worse."
  4. Everyone's pay get screwed up by alen · · Score: 4, Informative

    Army in the 90's
    Everyone's pay gets screwed up at least once, then. Uncle Sam takes it back

    Mostly minor things like an allowance like jump pay being paid while not on jump status

  5. Corruption by Anonymous Coward · · Score: 3, Insightful

    the Pentagon spent a billion dollars and wasn't successful

    Sounds like corruption. If you can't wrangle up a team of coders and project managers with a billion dollar carrot, then there is something terribly internally wrong with your process.

  6. How did the military pay during WW2? by Spy+Handler · · Score: 5, Insightful

    They had way more soldiers back then today, and payroll did not seem to be a problem. Maybe the Pentagon should go back to using whatever system they had back then.

    1. Re:How did the military pay during WW2? by Darinbob · · Score: 2

      Not true. Payroll software is amazingly complicated. And every year all the laws change so that you have to update it. It's constantly in flux. Federal, state, county, and sometimes even city laws to cover all the employees you have, plus of course foreign laws if you want to incorporate that into a single payroll department. That's why most places outsource this stuff. It's a huge expense to do it yourself and economies of scale make sense to have entire companies devoted to just that one job.

    2. Re: How did the military pay during WW2? by jfdavis668 · · Score: 2, Insightful

      The used 100 to 1000 times as many people. Working in un-air conditioned building doing routine work over and over. Then it was all double and triple checked. When they screwed up, you could yell at them. There is no way we could afford to recreate that system with today's pay scales. Also, these were sharp people. No one like that would even apply for the job. Welcome to the post industrial society.

  7. Re:Let me have a go at it.... by CrzyP · · Score: 2

    Give me a billion dollars and I can learn the process, learn TO code, code it, test it, and implement and support it myself.

  8. now there are multitude of pay levels. by Joe_Dragon · · Score: 3, Informative

    There is basic pay, plus “entitlements” for everything from serving in a combat zone to housing allowances to re-enlistment bonuses. An individual’s pay can change several times in a day.

    likely the old software can't deal with all of that to well.

    http://www.informationweek.com/government/state-local/outdated-it-blocks-california-payroll-or/225702383

  9. Can't read the article by Anonymous Coward · · Score: 5, Funny

    I tried to read the article, but it was written in English - a decades-old language.

    1. Re:Can't read the article by T.E.D. · · Score: 2

      The big problem with C11 is that is it no longer "backwards"-compatible with C++, so many will ignore it. For a large number of people "C" is going to remain defined as "the subset of C++ that I can get a C-only compiler to compile".

  10. Quick Books MIL by xdor · · Score: 2

    I wouldn't mind taking this one on. Sure it's already been done, but I'm sure they'd pay to have it done again.

    My only stipulation is that I'd want to do it all myself with just one business analyst and one quality control tester.

    I think I could manage it in 3 years at 3 million dollars. However I'd probably cause the loss of 10,000 jobs so its probably not going to happen :)

  11. Expecting too much by whizbang77045 · · Score: 2

    Aww, come on, fellas and gals: this IS the five sided puzzle palace we're talking about. You're expecting too much of them.

  12. 4.8 LOC per person by Anonymous Coward · · Score: 5, Interesting

    The code base is so large that its ~4.8 lines per active duty US military person. The code would be shorter if it was nothing but one line per person that prints how much to pay. You might argue that this would have maintenance issues, but automated porting of it would be trivial, and for the $billion they spent try to replace it, you could pay almost $1000 per print statement to keep it updated.

  13. Re:Redundant by Nerdfest · · Score: 3, Funny

    Also, seven million lines of COBOL is about what it takes to write a Sudoku solver in COBOL.

  14. Cobol really is self-documenting by techno-vampire · · Score: 3, Insightful

    Eh...there probably was some half baked documentation at some point,

    Yes, there was and there is. It's called "source code." One of the reasons that COBOL is such a verbose language is that it was designed so that bean counters with no programming experience could audit the source code and understand it well enough to make sure that nobody was stealing anything. Not only that, it's rare that COBOL code actually needs any comments because the variable names are long enough that you shouldn't ever have to guess what any of them are used for or what's being done with/to them.

    As far as spaghetti code goes, that can be a problem, especially in very old code, from before such things as structured programming were conceived. And, there's even a statement, "ALTER," which allows you to create self-modifying code, although even back in the '80s when I learned it in school, we were warned never to use it.

    --
    Good, inexpensive web hosting
    1. Re:Cobol really is self-documenting by Capsaicin · · Score: 5, Insightful

      Yes, there was and there is. It's called "source code."

      While it's true that COBOL is meant to be self documenting, there is, in a 7 million line project, a difference between understanding any particular section of code and being able to comprehend the entire structure of the project. If that structure is undocumented, you will have a lot of reading before you grasp the program globally. Apparently the "failures" that were being experienced were not leading the maintainers to the appropriate sections of code and such a global understanding had become necessary.

      I know it breaks one of the cardinal rules of software development, but if you have a cool billion to throw at the problem and the existing mess cannot be fathomed, perhaps starting afresh is an idea ...

      --
      Better to be despised for too anxious apprehensions, than ruined by too confident a security. --Edmund Burke
    2. Re:Cobol really is self-documenting by wallsg · · Score: 2

      Eh...there probably was some half baked documentation at some point,

      Yes, there was and there is. It's called "source code." One of the reasons that COBOL is such a verbose language is that it was designed so that bean counters with no programming experience could audit the source code and understand it well enough to make sure that nobody was stealing anything. Not only that, it's rare that COBOL code actually needs any comments because the variable names are long enough that you shouldn't ever have to guess what any of them are used for or what's being done with/to them.

      If I recall correctly from the CSC351 COBOL class I had as an undergrad in 1982 or 1983...

      COMPUTE t1(i1) = (t2(i2) ** 3) / (t3(i1) * 2 + 1);

      Completely made up, of course, but you can make "self-documenting" COBOL look nearly as bad as Perl if you want to.

    3. Re:Cobol really is self-documenting by techno-vampire · · Score: 3, Insightful

      COMPUTE t1(i1) = (t2(i2) ** 3) / (t3(i1) * 2 + 1);

      Yes, except that you'd have a period at the end, not a semicolon. (This is why statements in COBOL are referred to as "sentences.") However, if you did try something like that in a shop where the bean counters were auditing your code, they'd probably reject it for lack of clarity and insist that you gave that variable a more meaningful name. If you really want to, you can obfuscate code written in any language, but it's probably harder to get away with in COBOL than in most other languages.

      --
      Good, inexpensive web hosting
    4. Re:Cobol really is self-documenting by techno-vampire · · Score: 2

      There are very few non-programmers who can follow most source code and have the slightest idea of what's going on. In many cases, such as Perl, it's often not enough to be a programmer, you need to understand the specific language. COBOL is the only computer language I know of that can be followed by somebody with no programming experience at all.

      --
      Good, inexpensive web hosting
  15. Re:Outraged! by plover · · Score: 2

    contract out that stuff to someone who is good at it... like ADP.

    They did! That billion dollars wasn't to rewrite it -- that was just to buy and integrate a PeopleSoft package! Supposedly, PeopleSoft is good at that kind of thing, since that's what they sell to other big companies too stupid to write and maintain a simple payroll system.

    They don't need to try integrating to someone else's package again, unless they want to shovel another billion dollars into some other undeserving contractor's hands.

    --
    John
  16. call it what it is: fraud by markhahn · · Score: 2

    when the mil/gov spend a billion on some software project and it fails, we need to start calling it what it is: fraud perpetrated by consultant/contractors.

    it's bad enough when the industry burns 10-50M on an ERP project for a company (or university!), but pretty soon those tens of millions add up to real money. spending a billion should be HARD!

  17. Typical Government Efficiency by kartaron · · Score: 3, Informative

    For 33 years the government has been trying to replace the 60 year old air traffic control systems. Three different systems have been tried. The first was a complete write off, meant to be an IBM designed Unix based system, it went overdue by years and billions and was killed off in 1994. http://www.baselinemag.com/c/a/Projects-Processes/The-Ugly-History-of-Tool-Development-at-the-FAA/ The Second named CARTS began in 1996, meant to be a replacement for the aging radar systems the program did replace the older systems in some airports, but again the program was killed for cost overruns and stalled production. http://www.fiercegovernmentit.com/story/tracon-air-traffic-control-modernization-faces-prospect-more-schedule-cost/2013-06-02 http://www.bloomberg.com/news/2013-05-31/air-traffic-upgrade-over-budget-facing-delays-report.html In 2003 they revived the project with compartmentalized implementations of an integrated system in order to see short term improvements. The first system, a replacement for CARTS renamed STARS) went in in 2012 and it is costing 60% more than expected, with the remaining systems set to be developed and implemented over the next 13 years. The next system to be implemented, ERAM, is already overdue by 4 years, over budget, and according to FAA reports, subject to critical failures and instability. http://www.airtrafficmanagement.net/2013/06/nextgen-over-budget-delays-certain-report/http://www.fiercegovernmentit.com/story/eram-continues-undergo-critical-failures/2012-10-02 http://en.wikipedia.org/wiki/Next_Generation_Air_Transportation_System

  18. They might have lost the source by Trip6 · · Score: 2

    Very common in older mainframe shops. Poor source control. You can reverse engineer it but good luck.

    --
    I hate being bipolar; it's awesome!
  19. Re-write it in awk by russbutton · · Score: 2

    Cobol is based on the notion of processing what are essentially formatted text files, one line at a time. If you knew the format of a file to be read by a Cobol program, you could re-write it in awk and use about 1/50th the amount of code.

    Of course what really needs to be done is to document the actual process and system requirements, and then just put up a modern payroll processing system. The biggest problem is that there are a lot of people whose job it is to take a piece of paper from one bin and put it in another. All of those jobs would be put at risk were you to actually do something substantive about this problem. They're far more concerned with their jobs than actually getting anything done.

    1. Re:Re-write it in awk by DerekLyons · · Score: 2

      The biggest problem is that there are a lot of people whose job it is to take a piece of paper from one bin and put it in another. All of those jobs would be put at risk were you to actually do something substantive about this problem. They're far more concerned with their jobs than actually getting anything done.

      No, the problem is you don't understand the problem domain and thus just repeat platitudes on the assumption that doing so makes you look wise. This system has been in place for years, all the old paper movers are long gone.
       

      Of course what really needs to be done is to document the actual process and system requirements, and then just put up a modern payroll processing system.

      Everything *is* well documented - the problem is, the military pay system is way, way more complex than any civilian system. None of my old pay statements are handy (buried somewhere out in the garage if I still have them), but here's what went into my paycheck in the last year or so of my enlistment;

        • Base Pay: E-5 with over eight years of service.
        • Submarine pay. (But only so long as I was eligible for submarine duty - I was on shore duty, and if I'd gone under nine months with transferring to a sea command I'd no longer be eligible for that pay. This pay also varied with the number of years I'd been submarine eligible.)
        • COMRATS. Rather than issuing me a mess card to eat in the chow hall, I got cash instead.
        • BAQ - since I didn't live in the barracks, I got a cash allowance to live off base.
        • _____ I don't remember the name, but it was a differential to BAQ to account for the cost of living here. Someone in San Diego got more, someone in bumfuck Nebraska got less.
      • Then there were all the deductions (not so different from civilian practice, so no need for details.)
      • Then there were all the annual payouts, like my re-enlistment bonus and my uniform maintenance allowance.

       
      Had I gone back to sea, I'd have gotten sea pay - which amount depends on the total number of months served at a sea command during my career to date, so those months have to be tracked and the clock started and stopped as appropriate. (The services have a whole host of incentive, qualification, and duty status pay and allowances - jump, hazardous duty, onerous duty, etc... etc...)
       
      Then there's annual leave (which isn't so different from the civilian world.)
       
      On top of that, the system has to manage travel pay, the civilian pay system, retirees...
       
      It's a complex system, and nowhere as simple as just 'parsing a text file'.

  20. Re:Not to shocked by Areyoukiddingme · · Score: 5, Informative

    You are correct. And wrong. According to the Reuters article, there were more than 15,000 requirements changes during the lifetime of the project. So you had precisely the right idea. You just underestimated the ability of a bureaucracy to fight back. By an order of magnitude.

    And that's what it was, too. Make no mistake, the project failed because a successful software system would reduce the headcount of the DFAS dramatically. That couldn't be allowed to happen, so it was sabotaged by eternal feature creep.

    And of course, they started with PeopleSoft, and there's no organization better at absorbing all available money for no return besides the DOD itself. Talk about a match made in hell...