Glitches in Massive Government Databases?

HBergeron asks: "Rather then post this as another YRO in the litany of new government datamarts there is a more fundamental question for all the coding Slashdot readers out there. This story, in Government Executive magazine, outlines the range of programming glitches in what is a relatively simple database. As a matter of public policy (and taxpayer money) is this level of non-functionality to be expected in these sorts of projects? Is the contractor just ripping off the taxpayers with bad code? How hard is it to write software like this that works?" The article focuses on the SEVIS database, but have others noticed similar trend in other government information systems?

Comments

All software has bugs

[ObviousGuy]

And the government system of going with the lowest bidder is bound to cause some problems as the more expensive engineers would no doubt bring better experience and know how with them. When you bring in the inexperienced because they are cheap, you frequently end up spending more in the long run than if you had paid for the expertise up front.

It's like they say, you get what you pay for. Cheap prices are only cheap if your time has no value.

Re:All software has bugs

[Tablizer]

And the government system of going with the lowest bidder is bound to cause some problems as the more expensive engineers would no doubt bring better experience and know how with them. When you bring in the inexperienced because they are cheap...

Inexperience may be part of it, but often government systems are subject to a lot of competing interest and tying together existing diverse systems such that simple requirements in isolation often balloon into complicated situations. As a contractor your hands are often tied WRT cleaning up existing bad processes and odd requirements to solve needs of competiting agencies, departments, etc.

It is often diplomatic issues that cause the messes, not technical ones.

Re:All software has bugs

[aoteoroa]

To write perfect bug free software you must have a complete and accurate understanding of the end users problem. The best explanation I have found as to why this isn't as easy as it sounds was in a book called Software requirements and specifications which in the first chapter tells a story of a mathematician, finance director, manager, sociologist, and a stock broker discussing a recent failed project.

In 1993 the computer system project for the London Stock Exchange failed disastrously. 400 million pounds spent and nothing to show for it. Who was to pay? What had gone wrong? Why do so many developments end in disaster?

'Pure Ignorance,' said the mathematician. 'Software development is essentially a branch of mathematics. That is why computer science departments in universities have so often been closely associated with mathematics departments. You must understand that a program is a mathematical object. Its development is therefore a mathematical activity, of a particularly challenging kind. Those who engage in it should, of course, be competent both in using the appropriate mathematical notations and in drawing on the appropriate body of mathematical knowledge -that is, on knowledge of the relevant theorems. While we continue to ignore these facts we will continue to perpetrate disasters.'

'That's all very well,' said the finance director, 'but in my company we build systems to improve our business performance. I imagine that the Stock Exchange does the same. Software isn't mathematics: it's business. I think of a software development project as a capital investment. The test of its success is simply the value of the return on that investment to the company. The return in this case seems to be negative. The essential tools in a software project are financial risk analysis and discounted cash flow calculation'

'Of course you are right,' said the manager. 'But the key to achieving profitability and return on investment is to improve the development process, and with it the cost and quality of the end product. Software developers like to think they're doing something very special, but in fact it's an industrial process just like any other. The essence of software development is a quantitative approach to measuring and improving the performance of the software development process. What you don't measure you can't control.'

'But surely software development is done by people. And for people isn't it?' said the sociologist. 'Software is situated. You talk as if the system and its development were something objective, but really it has to be continually renegotiated subjectively between the various stakeholders, who all have their own agendas and perspectives. The success of any system depends directly on facilitating the negotiation, and on the determinant individual and group relationships in the societal context. I suspect that the Stock Exchange members belong to an authoritarian culture in which the dominant behaviour in inimical to peer group negotiation; perhaps that explains their failure.'

'This all seems ridiculous to me,' said the stockbroker. 'The plain fact is that the system was meant to serve the needs of brokers and jobbers of the Stock Exchange, and it didn't. It usually takes a professional working member of the exchange at least five years to learn how the Stock Exchange works, and I don't see why the analysts and programmers who make computer systems should expect to pick it up more quickly. A system for a particular business can only be built by people who are experts in that business. Domain knowledge, I think it is called. That's what matters.

Re:All software has bugs

[John Hasler]

> And the government system of going with the lowest
> bidder...

So you want them to give the contract to the most _highest_ bidder? The brother-in-law of the contract officer?

They give the contract to the lowest _qualified_ bidder. Doing otherwise would be stupid.

> It's like they say, you get what you pay for.

Bullshit.

Re:All software has bugs

[retto]

I've done some work with government organizations (not a lot so others way have had different experiences than mine) and from what I saw the biggest problem wasn't that the work was done by the lowest bidder, it was that the requirements were often created by people other than those that know the situation the best. Very little thought was given to designing something to be as usable and efficent as it can be, and more focus is given to making sure it gets finished by an arbitrary date. If it works, great, if not, it is ok as long as some department chief can say they are compliant with something by the required date. I've seen so many little problems that could have been fixed, but time involved in getting approval would have been more than then it was worth. I can't imagine the cumlative effect of all those little problems that get overlooked.

In the end you wind up with a big mess, tacking on or changing just enough to meet some kind of regulation. If you see something beyond the immediate scope of the project that would make things a lot easier and efficent, but it would require time/money or cooperation from someone else's department/division/agency it will be shot down as it won't be 'in the budget.'

Ok, that was my little rant. One time I had to sign a form to get the air conditioner turned on before the 'approved' time in a federal building, and I've been pissed off at bureaucracy ever since.

Re:All software has bugs

[homebru]

We build highways, bridges, and the like. We do the vast majority of our work under low bidder contracts. More importantly, we deliver the product on time and of a high quality.

And how do you deal with the customer whose specs say (in effect) "just throw a log across that creek because all we need is a footpath for the weekend" and subsequently declare your work an extension to the InterState Highway System and in non-compliance (substandard) of rule Blah, section Blah-blah, part Blah-blah-blah, paragraph Blah-blah-blah-blah?

From the article: The pilot was not designed to become a national system, however. The INS had intended to examine its results and then build something new, school officials who participated in the test say. It was a "throwaway project," says Johnson. "It wasn't supposed to become something bigger."

This is one of the most common causes of failure that I have seen over the years. A refusal by management to see the difference between a "proof of concept" project and a "production" project.

Attention programmers. Learn this now and learn it well. There is no such thing as a "quick and dirty" project. Anything you write for hire can and probably will be pushed into production. And if you "assumed" that you could "get by" with single user code with (for example) no record locking, error testing, logging, transactioning, or provision for remote monitoring or backup, you just screwed the pooch. The minute you check that code into CVS, it's heading for production with hundreds of incompetent users who will expect 100.000% uptime. And management will quickly point you out as the author of the failing new product and your reputation is shot, your future with the company is shot, and you have given programmers everywhere another black eye. Gee, thanks.

People, what it is, is that every piece of code that you write for hire has to be the very best you can create. Because, while your customer may have only asked you to throw a one-log footbridge across the creek, s/he is expecting an eight-lane interstate highway structure.

Re:All software has bugs

[HiThere]

The problem is probably worse than just inexperience. I don't know what it is, and I work in the middle of it. Probably it's empire-building, or at least that's my guess.

Example: I have the database that I put together for my (small agency). It took a few months to do, but not many... perhaps three, and that's allowing for a bunch of mistakes. Then it was refined for a few years.

Three years ago we were told that a State Agency would be assuming the duty. They were legally obligated to begin about a year ago. They still aren't doing it. But I built it myself, but when I talk to the State people:
1) it's a team of people building it
2) the people don't stay at the same job
3) sometimes it's being done by people in a different department.

When I went to give them my data, they wouldn't even take an electronic file. They wanted typed forms. (I cheated and printed them out on a laser printer...but the spacing had to match that of a typewritten form.)

Now this was about 4,000 sheets of paper that they were going to need to enter by hand. My *guess* is that they were entering it into a 3270 based system (i.e., old IBM mainframe) and were then going to convert it into a PC based system. They were implementing it in FoxPro...probably. I don't really know. I never ended up talking to anyone who was doing the real work.

Personally, this was what I consider my "small test system". It's the one I traditionally converted from system to system during the process of learning the new system. I'm told that they'll be on-line with it in a month. (They'd better hurry, or I'll retire before they get it running... not that it matters.)

So I sure understand about the slowness, and I have a few theories as to what causes it: You need to take the cross-product of the Parkinson Laws with the "Mythical Man-Month". Then add a bunch of Dilbert for seasoning. Gourmet crusine to revolt the finest palate.

There are times when this is a good thing

[KPU]

If the government has a harder time keeping track of people, maybe it will be less ambitious. Never mind.

Surprising?

[bajo77]

This seems to be on par with other things the government tries to keep tabs on. They can't keep track of paroled felons, the database of people who can't vote is horribly flawed, and the soundex database that the airlines use doesn't work either.

Granted, this needs to change, but this isn't the first time the government has failed to provide adequate information regarding lists of people.

Taxpayer $?

[spumoni_fettuccini]

Is the contractor just ripping off the taxpayers with bad code?

They've been doing that for years. Toilet seats for $10,000; hammers for $7,000? Not only that how much money is wasted on the old "that's exactly what I asked for, but not what I wanted"?

Re:Taxpayer $?

[Brymouse]

You forget the 10,000 toliet seat was designed for use on bombers. You can use the toliet in sever turblance, and if the plane flips, you will not get shit all over yourself. In a combat situation, this is a bargian at 10k. Imagian if your taking a shit and you come under fire, with a convetional airline toliet you would be busy cleaning up, now that could cost you your life.

EDS? Explains a lot...

[NickFitz]

EDS do a lot of systems that don't work, or don't work properly, or run massively over schedule and budget, here in the UK as well.

I just can't understand why governments insist on using them with the track record of cock-ups they have; they're not even cheap.

25 Years of Government

[Grey_Coder]

I have been working for municipalities for 25 years. I have yet to see a major program work well or work at all without overruns. I have chalked it up to me lacking a MBA or Degree in Computer Science. I am just a poor hobbiest that thinks for a million or three you should get what you pay for. But like shrinked software there must be no implied warrantee or garentee it will work. Man I think for a couple million give me a couple coders and little hardware and sit back. Open source here we come.

'Lowest Tender' syndrome.

[The Famous Druid]

I've seen this sort of thing happen before.

Government departments are pretty-much obliged to go with the lowest tender, even if the people running the tender know that the winning bidders are a bunch of incompetents who couldn't organize a fsck in a brothel.

So, the lowest bid wins, and then even if they actually are well-meaning and try to do the right thing, they have such limited resourses that they usually have to resort to working too few staff too many hours.

The result will not be quality code.

Not suprising.

[Mr. Piddle]

How hard is it to write software like this that works?

Harder than you imagine. If you remove the pork-barrel politics, directives of what technology to use coming from the clouds, and the recently potty-trained project team members, there isn't much left to give the project a chance at success. Most of the project's time is probably spent learning the difference between JDBC and EJB or at meetings discussing the differences between JDBC and EJB. The remaining time is spent accomplishing little by discussing the well-presented but vacuous system requirements for the project. Whenever I see a job posting for a database project for a government agency, I pass it and look for projects worth doing. If it mentions .NET or J2EE, I pass it by doubly fast.

I don't like this conclusion, but I've worked on, interviewed for, or heard about enough of these projects to realize they are pretty much all the same and for what seems to be all the wrong reasons.

its about "now"

[cr@ckwhore]

I have first hand experience with this subject after spending 2 long years working with a State level government agency to develop motor vehicle registration software ...

The problem is not so much about "how hard is it to write software that works" ... its more about "we're writing software for what we need RIGHT NOW".

When governments sit down to write software, its usually done through private contractors. So, a group of beaurocrats have a pow-wow and come up with a spec that generally reflects the type of work that the agency is doing "now", without much future consideration.

15 years later ... as legislation, beaurocracy, and agency regulations expand, so do the requirements of the software. For example, the Bureau of Motor Vehicles in an unspecified state put their first computer system in place in 1968. Since then, the scope of the BMV has expanded at least 10-fold.

Complicating the issue, "upgrades" are usually in the form of applying a new "layer" to the system somehow. As of 2003 in this unspecified state, the typical motor vehicle registration passes through 4 different systems before arriving in the central (OLD and limited) database at the state.

Complicating the problems even further are the many new layers of regulatory bloat -- meaning, the BMV is using software that met their needs in 1968, but doesn't meet their needs now. For example, (and this is how data goes bad), they're required to track whether or not somebody's registration is under suspension. However, back in 1968 registration suspension wasn't even a blip on the radar. To handle the problem after the "registration suspension legislation" was enacted, an "exception" had to be built into the system... if the street address field contains a special message, it indicates that the registration is under suspension. Ultimate problem... fields in the database are being used for purposes they were never intended. The age of the system does not allow for it to be updated properly.

Funny how you never hear ...

[Professor D]

The conspiracy theorists talk about how damned inefficient, bloated, clumsy and self-defeating government agencies projects are.

Somehow "they" have had UFO technology which would make petroleum obsolete since the '50s, conspired to kill JFK to keep it a secret, brainwashed Chapman to murder Lennon, created a secret government database tracking everyone's cash transactions, control us by putting chemicals in our water and thought patterns in satellite broadcasts. Oh yeah and "they" also were behind the 9/11 attacks as well.

Yet "they" can't even figure out how to keep track of whether or not foreign students went to class or not.

MS BS

[coyote-san]

That's MS BS. (And the cry of incompetent programmers for decades.) Even if we agree that all software has bugs - and I don't - that canard says nothing about all bugs being equal much less anything about all software having about the same number of bugs.

Any competent manager would know that experienced coders are usually FAR cheaper than inexperienced ones because they make fewer mistakes due to ignorance or indifference ("it works for me, so it's done!"). That gets you to the point of dealing with the more subtle and intrinsic bugs (e.g., due to conflicting requirements) quicker and cheaper, and the apparent cheaper cost of inexperienced developers is only achievable if you plan to release after coding is finished, not after testing is completed. Which is pretty much every MS *.0 release, now that I think about it -- got to get to market first, even if it's pure crap!

And another thing...

[Mattcelt]

I think there's an even darker side than what's being presented here in the brief - consider what happens when one of these 'glitches', whether techinical or PEBKAK, cause inaccurate information to be propogated through the linked government databases such as the TIA? Does joe traveller get strip-searched at every airport he goes to because someone "accidentally" put his name onto a terrorist watch list? Where does the government's responsibility to be accurate and precise with our information end? If the Credit Reporting Agencies are any indication, I think we have a potentially huge problem on our hands.

System doesnt work

[t0ny]

I work with the government as a consultant, and I continue to be amazed at the waste and stupid use of money involved in government projects. What seems to be the rule, rather than the exception, is poorly skilled people pitching initiatives they have absolutely no skills to impliment, and they dont even put in the work to design it well.

There is one database used for payroll on which millions (if not tens of millions) were spent, and the end result is a system nobody is completely sure of, and which requires all the deparments involved to completely change all their procedures. And its even less flexible and problematic than what it is replacing. AND this is a custom application!

Until government starts paying tech people what they are worth in the private sector, you will always have poorly skilled bullshitters pedalling their wares to the public sector, who is suffering from the illusion that throwing money at a problem will make it go away.

Re:Anyone seen "Brazil"?

[qtp]

what's frightening about Ashcroft is not that he's a fundamentalist autocrat, but that he's an incompetent fool.

The real danger of John Ashcroft is that he's a fundamentalist autocrat and an incompetent fool.

Bureaurcy and the lowest bidder

[bastion]

As a matter of public policy (and taxpayer money) is this level of non-functionality to be expected in these sorts of projects?

Yes.

Look on the bright side, if they can screw up simple projects like this how far do you really think TIA (Total Infomation Awareness) is gonna get?

TIA Alias Search: Commander Taco

Output: Mexican terrorist. Leader of collbration site for social dissidents.

Just far enough from the truth to get somebody sued....

All software has bugs

[Tony-A]

The thing is to be aware of this before coding and take steps to compensate and reduce the damage from those bugs.
An excuse after the fact, the canard seems infantile.

All software has bugs. They are most certainly not created equal. Just because there is no way to expose them, doesn't mean they're not there.

("it works for me, so it's done!")
OUCH!
There is a big difference between partially working for a few things for a few people and never failing for everything for everybody. Some of the critical knowledge lies in the application area and is neither achieved readily or even necessarily consistent.

Brief Rebuttal

[Bios_Hakr]

I have been working in the USAF for about 8 years. 6 of that in WAN (longhaul voice and data), and 2 in Infrastructure and security. I'd like to offer another side to your story:

>but it needed to be done before the end of the fiscal year

This is how it works: The USAF has a budget. Each area gets a small slice. If filters down to each office having about $10k ~ $30k for operations that year. That money has to last all year. About 20% of that is kept in reserve funds. If that money is not needed by August, we are free to spend it. At that point, we develop a wish list and try to get that aproved. By time all this happens, we have about 5 weeks to spend the reserve money.

No one in the military likes it. All our contractors hate it. If you want it changed, write your congressperson and have them change 50+ years of bad management practices...

>The quality of code would've been greately improved if we coded, say 40 hrs/week instead of pulling all-nighters.

I have spent countless days and nights working overtime. So have a lot of my coworkers. In times of exercise or, God forbid, a war, we go to 12+ hour days. 6 days on and 1 day off are common during exercises.

Contractors always make fun of us for sloppy wiring, half-assed installs, unpatched servers, etc... When new equipment arrive, we usually have a few hours to determine where it will go and when. We are usually told that the old equipment stays in place until the new stuff is operational. This leads to massive misuse of rack space. and cluttered wiring.

Also, just like your code suffers from 40+ hours, my wiring suffers when I have to spend my Saturday morning connecting a new router.

No one likes to work overtime. Your work suffers just like mine. You may lose a contract because of your bad code. People could lose lives because of my bad wiring. Let's both work harder to keep our shit straight, regardless of hours worked.

>They insisted on .NET 2003 server with M$ SQL, etc., etc.

This is becuase we have a very nice license with MS for their stuff. We get good support, including semi-annual "Best Practices" reviews by MS inspectors. The US Gov paid for MS tools, we should use them. If you don't like it, write your congressperson. Personally, I'd love to be able to use Squid on Red Hat. Unfortunately, we don't have the money to spend on more software licenses after we bought MS stuff.

>asked if it would work with a win2003 server as opposed to a win2k

Our upgrade paths are fixed by MS. This absolutely sucks. Our systems require specific patch releases from MS. Once they stop supporting those patch paths, we have to upgrade. Agian, if you don't like it, write your congresscritter.

>but they didn't even know how to install windows

I'm throwing a bullshit flag on this play. I find it difficult to belive that no one knew how to install Windows. In the USAF, we have a NCC department that does nothing but install, configure, and maintain Win2k servers.

There may have been an internal power play based on getting Win2k3 server training. That is an ongoing military issue. Your boss tells you to do something. If you do it and screw it up, they ask if you were trained to do that thing. If you were not trained, then you go to federal-pound-me-in-the-ass prison for working on something without proper training. If you were trained and you screw it up, then you get in trouble for not folowing the training guidelines for whatever it was you broke.

Everyone working in a military NCC can install Win2k Workstation and Server. Many of them are MSCEs or higher. They could probably install Win2k3. They just wanted official training on that product before they tried something and broke it.

>Installing a a windows server is a mountain of a task for them.

No it isn't.

>Installing .NET is something that, as they say, they 'have been working on for

It's 'Unit Tests', stupid!

[adamscottphotos]

I'm amazed that I haven't seen a single poster bring this up yet so...

The key to quality software; flexible, extensible, fault-tolerant, maintainable, and all of those other adjectives that 'good' software is supposed to have is very, very simple.
It's called Unit Testing. It's not brain surgery. I have worked on several medium to large scale projects (500k-3.5m lines) in several languages and environments, and I've yet to bomb one hard using this methodology, despite the usual client shenanigans.

Every time I write a functional subunit, I start by writing a series of tests (based on the spec, hopefully) that define 'doneness' for that subunit. Every object in the system has it's own set of tests. The test harnesses are chained together, so I can hit a button, so to speak, and run all of the hundreds to thousands of tests at once.

Whenever I check in new code changes, I run the test suite. If a test fails that previously worked, then I broke something. This plus good OOP practices (low coupling, high cohesion) allows you to make changes on the fly without the kind of 'The Money Pit' syndrome (fix one thing, another breaks) that is described in the article.

I am certain that the system in question was NOT developed with these methods. Most development organizations that I come into contact with pay lip service to the concept, but don't want to spent the perceived extra $ up front. The thought of all those developers writing TESTS when they could be writing CODE scares the willies out of them. But it pays for itself. It really does, every single time. Don't tell your boss, and try it on your next project. This is old news - google has a ton of info on it, and there are some good but unnecessary books also.

In the dot.com glory days, we had a huge system, running several hundred transasctions per second on a geographically distributed system of clients. We made fundamental architectural changes without a hitch, switched servers live without a hitch etc, and made a zillion little changes, all live, and all without a hitch (well, other than really stupid human errors, like locking out the client upgrade system with a bad password... oops). We had zero budget, and 2.5 developers. Unit tests are the Way, and if any company that doesn't mention them in your first meeting, run like hell.

Reactive, Just-In-Time Development

[prestidigital]

I've read article and many of the replies here. Several /.ers definitely describe flaws in government contracting processes and hiring practices that I've experienced, but I think they are missing the point. I think there is an additional, more fundamental flaw, that has been overlooked - or at least didn't get modded up high enough for me to see - maybe i should go trolling for a different set of opinions :^).

My experience tells me that the problems begin when we fall into the trap of trying to solve problems with a reactive mindset instead of a proactive mindset (proactive being favorable). We allow daunting problems and/or a need for revenue to back us into a corner time and time again and every time we are forced to hack our way out. Some of that is just old-fashioned survival, but a lot of it can be avoided with deliberate forethought, planning, discipline, and a commitment to quality and detail.

Avoiding clusterf*x has to be an institutional effort, whether the institution is a huge goverment agency or an tee-tiny, independent software shop. Everyone in the organization - operations, sales, IT - has to be on board with the policy that "if it's worth doing at all, then it's worth taking the time to do it right...the first time." I said "fundamental" earlier b/c that has to be something like lesson #5 in life's little handbook - we all heard it all too often when we were kids, we know it's true, and yet now we don't pay it any mind.

I do think the failure to heed that simple maxim usually starts in business development and snowballs by the time it gets to IT, but it really goes both ways. Everyone has to be responsible for maintaining the discipline required to produce quality.

What happened with this system is everyone involved got themselves all in a panic like a drowners who not only won't let you save them, but pull you under too. It's understandable given Sept 11, but "undertandable" and "right" are two different things. Legislators threw money at a situation they didn't try to understand. Deal-makers when after that money, promising to solve problems they knew they didn't understand. Developers enabled deal-makers by claiming to understand. No one took the time to do it right the first time.

P.S. It doesn't have to be this way.