Glitches in Massive Government Databases?

HBergeron asks: "Rather then post this as another YRO in the litany of new government datamarts there is a more fundamental question for all the coding Slashdot readers out there. This story, in Government Executive magazine, outlines the range of programming glitches in what is a relatively simple database. As a matter of public policy (and taxpayer money) is this level of non-functionality to be expected in these sorts of projects? Is the contractor just ripping off the taxpayers with bad code? How hard is it to write software like this that works?" The article focuses on the SEVIS database, but have others noticed similar trend in other government information systems?

Comments

All software has bugs

[ObviousGuy]

And the government system of going with the lowest bidder is bound to cause some problems as the more expensive engineers would no doubt bring better experience and know how with them. When you bring in the inexperienced because they are cheap, you frequently end up spending more in the long run than if you had paid for the expertise up front.

It's like they say, you get what you pay for. Cheap prices are only cheap if your time has no value.

Re:All software has bugs

[Tablizer]

And the government system of going with the lowest bidder is bound to cause some problems as the more expensive engineers would no doubt bring better experience and know how with them. When you bring in the inexperienced because they are cheap...

Inexperience may be part of it, but often government systems are subject to a lot of competing interest and tying together existing diverse systems such that simple requirements in isolation often balloon into complicated situations. As a contractor your hands are often tied WRT cleaning up existing bad processes and odd requirements to solve needs of competiting agencies, departments, etc.

It is often diplomatic issues that cause the messes, not technical ones.

Re:All software has bugs

[aoteoroa]

To write perfect bug free software you must have a complete and accurate understanding of the end users problem. The best explanation I have found as to why this isn't as easy as it sounds was in a book called Software requirements and specifications which in the first chapter tells a story of a mathematician, finance director, manager, sociologist, and a stock broker discussing a recent failed project.

In 1993 the computer system project for the London Stock Exchange failed disastrously. 400 million pounds spent and nothing to show for it. Who was to pay? What had gone wrong? Why do so many developments end in disaster?

'Pure Ignorance,' said the mathematician. 'Software development is essentially a branch of mathematics. That is why computer science departments in universities have so often been closely associated with mathematics departments. You must understand that a program is a mathematical object. Its development is therefore a mathematical activity, of a particularly challenging kind. Those who engage in it should, of course, be competent both in using the appropriate mathematical notations and in drawing on the appropriate body of mathematical knowledge -that is, on knowledge of the relevant theorems. While we continue to ignore these facts we will continue to perpetrate disasters.'

'That's all very well,' said the finance director, 'but in my company we build systems to improve our business performance. I imagine that the Stock Exchange does the same. Software isn't mathematics: it's business. I think of a software development project as a capital investment. The test of its success is simply the value of the return on that investment to the company. The return in this case seems to be negative. The essential tools in a software project are financial risk analysis and discounted cash flow calculation'

'Of course you are right,' said the manager. 'But the key to achieving profitability and return on investment is to improve the development process, and with it the cost and quality of the end product. Software developers like to think they're doing something very special, but in fact it's an industrial process just like any other. The essence of software development is a quantitative approach to measuring and improving the performance of the software development process. What you don't measure you can't control.'

'But surely software development is done by people. And for people isn't it?' said the sociologist. 'Software is situated. You talk as if the system and its development were something objective, but really it has to be continually renegotiated subjectively between the various stakeholders, who all have their own agendas and perspectives. The success of any system depends directly on facilitating the negotiation, and on the determinant individual and group relationships in the societal context. I suspect that the Stock Exchange members belong to an authoritarian culture in which the dominant behaviour in inimical to peer group negotiation; perhaps that explains their failure.'

'This all seems ridiculous to me,' said the stockbroker. 'The plain fact is that the system was meant to serve the needs of brokers and jobbers of the Stock Exchange, and it didn't. It usually takes a professional working member of the exchange at least five years to learn how the Stock Exchange works, and I don't see why the analysts and programmers who make computer systems should expect to pick it up more quickly. A system for a particular business can only be built by people who are experts in that business. Domain knowledge, I think it is called. That's what matters.

Re:All software has bugs

[vladkrupin]

The quote from the original article:
How hard is it to write software like this that works?

Wow! Well said! My grandma couldn't have done better. In other words, please define 'works' for me. How many blue screens a day constitute 'works' and how many are too many?

Also, since we are at it, I want to reflect back on the latest project we have done. Incidentally, for the government. Before asking if a vendor is ripping the taxpayer off we need to consider how the government mismanages the resources it has. Consider the facts:

1. The project itself was fairly small and simple. I'd say it would normally take about 2 months to develop and deploy, but it needed to be done before the end of the fiscal year, so it was a 'now-or-never' situation, and was a horrible time-crunch. We had slightly more than half the time necessary to do it, but they won't even try to install it till probably the end of the year! The quality of code would've been greately improved if we coded, say 40 hrs/week instead of pulling all-nighters.

2. They tried to keep tabs on the development by scheduling 'technical meetings' over the phone. While there is nothing wrong with that per se, in a time-crunch that was a horrible waste of time. The smartest things we've heard from them were questions like 'Are you using hungarian notation?' or 'is your code well-documented?'....

3. They insisted on .NET 2003 server with M$ SQL, etc., etc. We did our best to make them consider PHP and the like, because that's what we normally use, but they were willing to pay extra to have that stuff developed in all-M$ stuff! We were told that the reason for that was because their IT was managing only M$ software, and the server was already there, and they couldn't have anything else (e.g. PHP). Fine, I can understand some bureaucracy in IT - that's cool, but imagine my surprise when, after we shipped them a CD with the project, they called us back and asked if it would work with a win2003 server as opposed to a win2k!!! Not only they didn't have the server yet, (or the infrastructure for that matter), but they didn't even know how to install windows! Which brings me to point #4:

4. Their IT is kick-ass. As in 'their ass needs to be kicked real hard'. Installing a a windows server is a mountain of a task for them. Installing .NET is something that, as they say, they 'have been working on for a while, but haven't got it quite yet'! And, when we give them a database dump they have no idea what to do with it and you have to walk them through the process (right-click on the 'Databases', select 'Create New database', click ok...) And they are paying these people!!! Errr... Let me re-phrase that - We are paying the government to employ those dumbheads! Thatnks goodness the network on which that is installed is not connected to the internet - the same idiots are in charge of security as well.

Yes, it is true that some contractors will rip off the government (and it is really the government's responsibility to make sure that doesn't happen! But that's not the point). The point is that even if they have a perfectly good product developed by honest people, they are still remarkably talented at screwing it up. Bureaucracy and lots of idiots in charge of hiring people are to blame.

Re:All software has bugs

[hendridm]

> And the government system of going with the lowest bidder is bound to cause some problems

I worked in a state agency, and the fact that we were required to take bids didn't really change who we purchased from. We just chose the vendor we liked best and justified it by writing the project needs around that vendor. They did that with employees too. When a new job opened, they often had someone in line for the position. However, equal opportunity required that the do interviews for the position. To justify the person they desired, they would write the job description and requirements based on the skills of the individual they liked. They would then schedule interviews even though they already had someone chosen for the position, just to meet requirements. I suppose they could have changed their mind if they found someone who was absolutely fabulous, but it's hard to convince an employer how great you are when in the back of their mind they don't think the interview is going to matter anyway.

Re:All software has bugs

[homebru]

We build highways, bridges, and the like. We do the vast majority of our work under low bidder contracts. More importantly, we deliver the product on time and of a high quality.

And how do you deal with the customer whose specs say (in effect) "just throw a log across that creek because all we need is a footpath for the weekend" and subsequently declare your work an extension to the InterState Highway System and in non-compliance (substandard) of rule Blah, section Blah-blah, part Blah-blah-blah, paragraph Blah-blah-blah-blah?

From the article: The pilot was not designed to become a national system, however. The INS had intended to examine its results and then build something new, school officials who participated in the test say. It was a "throwaway project," says Johnson. "It wasn't supposed to become something bigger."

This is one of the most common causes of failure that I have seen over the years. A refusal by management to see the difference between a "proof of concept" project and a "production" project.

Attention programmers. Learn this now and learn it well. There is no such thing as a "quick and dirty" project. Anything you write for hire can and probably will be pushed into production. And if you "assumed" that you could "get by" with single user code with (for example) no record locking, error testing, logging, transactioning, or provision for remote monitoring or backup, you just screwed the pooch. The minute you check that code into CVS, it's heading for production with hundreds of incompetent users who will expect 100.000% uptime. And management will quickly point you out as the author of the failing new product and your reputation is shot, your future with the company is shot, and you have given programmers everywhere another black eye. Gee, thanks.

People, what it is, is that every piece of code that you write for hire has to be the very best you can create. Because, while your customer may have only asked you to throw a one-log footbridge across the creek, s/he is expecting an eight-lane interstate highway structure.

Surprising?

[bajo77]

This seems to be on par with other things the government tries to keep tabs on. They can't keep track of paroled felons, the database of people who can't vote is horribly flawed, and the soundex database that the airlines use doesn't work either.

Granted, this needs to change, but this isn't the first time the government has failed to provide adequate information regarding lists of people.

Government Waste

[simsj]

This make me glad I don't pay taxes

Had to.

[Valar]

Neo: I just had a deja vu.
Morpheus: What? What did you see?
Neo: I saw the same Bush pass by twice.
Morpheus: Was it exactly the same Bush?
Neo: I dunno... could've been some kind of father son thing.
Morpheus: A deja vu is a glitch in the database. It usually happens when they change something. Particularly, votes.

25 Years of Government

[Grey_Coder]

I have been working for municipalities for 25 years. I have yet to see a major program work well or work at all without overruns. I have chalked it up to me lacking a MBA or Degree in Computer Science. I am just a poor hobbiest that thinks for a million or three you should get what you pay for. But like shrinked software there must be no implied warrantee or garentee it will work. Man I think for a couple million give me a couple coders and little hardware and sit back. Open source here we come.

This isn't really news

[dirtfirst]

...Except perhaps to the executives the magazine is aimed at. Early versions of software are generally pretty buggy, particularly if the target keeps changing, and most especially if it is in response to a hastily crafted law. The only thing that's surprising about this is that the output is taken so seriously by law enforcement officials *prior to completion*.

Don't they have some donuts that need eating?

its about "now"

[cr@ckwhore]

I have first hand experience with this subject after spending 2 long years working with a State level government agency to develop motor vehicle registration software ...

The problem is not so much about "how hard is it to write software that works" ... its more about "we're writing software for what we need RIGHT NOW".

When governments sit down to write software, its usually done through private contractors. So, a group of beaurocrats have a pow-wow and come up with a spec that generally reflects the type of work that the agency is doing "now", without much future consideration.

15 years later ... as legislation, beaurocracy, and agency regulations expand, so do the requirements of the software. For example, the Bureau of Motor Vehicles in an unspecified state put their first computer system in place in 1968. Since then, the scope of the BMV has expanded at least 10-fold.

Complicating the issue, "upgrades" are usually in the form of applying a new "layer" to the system somehow. As of 2003 in this unspecified state, the typical motor vehicle registration passes through 4 different systems before arriving in the central (OLD and limited) database at the state.

Complicating the problems even further are the many new layers of regulatory bloat -- meaning, the BMV is using software that met their needs in 1968, but doesn't meet their needs now. For example, (and this is how data goes bad), they're required to track whether or not somebody's registration is under suspension. However, back in 1968 registration suspension wasn't even a blip on the radar. To handle the problem after the "registration suspension legislation" was enacted, an "exception" had to be built into the system... if the street address field contains a special message, it indicates that the registration is under suspension. Ultimate problem... fields in the database are being used for purposes they were never intended. The age of the system does not allow for it to be updated properly.

Funny how you never hear ...

[Professor D]

The conspiracy theorists talk about how damned inefficient, bloated, clumsy and self-defeating government agencies projects are.

Somehow "they" have had UFO technology which would make petroleum obsolete since the '50s, conspired to kill JFK to keep it a secret, brainwashed Chapman to murder Lennon, created a secret government database tracking everyone's cash transactions, control us by putting chemicals in our water and thought patterns in satellite broadcasts. Oh yeah and "they" also were behind the 9/11 attacks as well.

Yet "they" can't even figure out how to keep track of whether or not foreign students went to class or not.

Q:Elepehant? A:Mouse built to government spec.

[blhseawa]

This is the same old software engineering problem, over and over again.

A user who has never written a *COMPLETE* system specification, acutal has no idea what that is, who only knows what he/she does not want.

Software developers/coders/bodies who are not SME's (subject matter experts), making system / software decisions without either the knowledge or guidance to understand the ramifications of those decisions.

Neither users, nor software development companies want to deal with these issues, they would rather just get the money.

That is why most large software development/ service companies have such bad reputations.

According to SEI, (Software Engineering Insitute) over 70 per cent of all software development projects are terminated as failures.

The key is in the rate structure

[plsuh]

I've worked on some government contracts, and in my opinion a big part of the problem is in the GSA schedule rate structure that the Federal uses for contractors. It is much more profitable for a contractor on a government project to put many junior people on a project rather than a few senior people, for the same amount of revenue. For instance, a junior developer may cost a contractor $50/hr with overhead, but the contractor is able to bill the government for that junior developer at $150/hr., a spread of $100/hr. A senior developer may cost $100/hr with overhead, but can only be billed to the government at $175/hr, a spread of only $75. Furthermore, the contractor can bill more hours of junior time than senior time under a given budget cap, compounding the effects of the greater spread. Thus, the incentives for contractors are to use as many junior developers as possible on a project, to increase the profit margin.

Unfortunately, It's a rule of thumb in this industry that a few good programmers are a lot more productive than many unskilled ones. The result is that many government IT projects are shoddily built by well-meaning but inexperienced developers who are put in that position by a contracting structure that fails to recognize the realities of the IT industry. Contractors are just responding rationally to the incentives that are presented to them.

These numbers are examples -- in fact the situation may be even worse. Federal government contracts vary in their rate structures, and many are stingier than this. It may well be impossible to bring on a senior developer as a subcontractor because the maximum hourly rate that the government will pay on a project is lower than the cost of the senior developer.

A prime contractor that I worked with staffed a large WebObjects project for the Department of Defense with a dozen or so low-paid, fresh out of community college drones. Every six months -- when a project review was due -- they would bring us on board as subcontractors for six to eight weeks. In that time, two or three of us would take the code base from where it was four months ago and bring it close enough to the required progress to get the contract renewed, and then the prime contractor would say "goodbye" and toss us out. Four months or so would pass by, with their people making little meaningful progress, and we would get a panicked call for six or eight weeks of more work to get by the next project review. (Did I mention that the prime contractor didn't pay the bills for one set of work until they needed us for the next project review? It got so bad that at one point we had to treat them as though their credit rating was zero, and demanded that payment for each week's worth of work be deposited in an escrow account before we would continue.)

By the way, this rate structure is not unique to government IT projects. Other types of government projects display the same professional services rate structure. When I worked for a (then) Big Six accounting firm as an economist, most consulting projects for corporate clients were staffed with a ratio of one partner and two or three senior managers to six or eight associates. However, the Federal government group was staffed with a ratio of one partner and one senior manager to twenty or so associates. I talked to the senior manager, and he told me that (a) the associates in the government contracting group were paid much less than we were on the corporate side since they billed out at a lower rate, and (b) the only way they could make money was to use lots of cheap associates because senior people could only break even at best at government rates.

Ya know, it'd be nice to see a GSA person squirm over this sort of thing in front of Congress some time. Then again, Congress may be part of the problem, as they'd rather generate lots of jobs for constituents, instead of a few.

--Paul

Brief Rebuttal

[Bios_Hakr]

I have been working in the USAF for about 8 years. 6 of that in WAN (longhaul voice and data), and 2 in Infrastructure and security. I'd like to offer another side to your story:

>but it needed to be done before the end of the fiscal year

This is how it works: The USAF has a budget. Each area gets a small slice. If filters down to each office having about $10k ~ $30k for operations that year. That money has to last all year. About 20% of that is kept in reserve funds. If that money is not needed by August, we are free to spend it. At that point, we develop a wish list and try to get that aproved. By time all this happens, we have about 5 weeks to spend the reserve money.

No one in the military likes it. All our contractors hate it. If you want it changed, write your congressperson and have them change 50+ years of bad management practices...

>The quality of code would've been greately improved if we coded, say 40 hrs/week instead of pulling all-nighters.

I have spent countless days and nights working overtime. So have a lot of my coworkers. In times of exercise or, God forbid, a war, we go to 12+ hour days. 6 days on and 1 day off are common during exercises.

Contractors always make fun of us for sloppy wiring, half-assed installs, unpatched servers, etc... When new equipment arrive, we usually have a few hours to determine where it will go and when. We are usually told that the old equipment stays in place until the new stuff is operational. This leads to massive misuse of rack space. and cluttered wiring.

Also, just like your code suffers from 40+ hours, my wiring suffers when I have to spend my Saturday morning connecting a new router.

No one likes to work overtime. Your work suffers just like mine. You may lose a contract because of your bad code. People could lose lives because of my bad wiring. Let's both work harder to keep our shit straight, regardless of hours worked.

>They insisted on .NET 2003 server with M$ SQL, etc., etc.

This is becuase we have a very nice license with MS for their stuff. We get good support, including semi-annual "Best Practices" reviews by MS inspectors. The US Gov paid for MS tools, we should use them. If you don't like it, write your congressperson. Personally, I'd love to be able to use Squid on Red Hat. Unfortunately, we don't have the money to spend on more software licenses after we bought MS stuff.

>asked if it would work with a win2003 server as opposed to a win2k

Our upgrade paths are fixed by MS. This absolutely sucks. Our systems require specific patch releases from MS. Once they stop supporting those patch paths, we have to upgrade. Agian, if you don't like it, write your congresscritter.

>but they didn't even know how to install windows

I'm throwing a bullshit flag on this play. I find it difficult to belive that no one knew how to install Windows. In the USAF, we have a NCC department that does nothing but install, configure, and maintain Win2k servers.

There may have been an internal power play based on getting Win2k3 server training. That is an ongoing military issue. Your boss tells you to do something. If you do it and screw it up, they ask if you were trained to do that thing. If you were not trained, then you go to federal-pound-me-in-the-ass prison for working on something without proper training. If you were trained and you screw it up, then you get in trouble for not folowing the training guidelines for whatever it was you broke.

Everyone working in a military NCC can install Win2k Workstation and Server. Many of them are MSCEs or higher. They could probably install Win2k3. They just wanted official training on that product before they tried something and broke it.

>Installing a a windows server is a mountain of a task for them.

No it isn't.

>Installing .NET is something that, as they say, they 'have been working on for