Glitches in Massive Government Databases?

HBergeron asks: "Rather then post this as another YRO in the litany of new government datamarts there is a more fundamental question for all the coding Slashdot readers out there. This story, in Government Executive magazine, outlines the range of programming glitches in what is a relatively simple database. As a matter of public policy (and taxpayer money) is this level of non-functionality to be expected in these sorts of projects? Is the contractor just ripping off the taxpayers with bad code? How hard is it to write software like this that works?" The article focuses on the SEVIS database, but have others noticed similar trend in other government information systems?

Comments

All software has bugs

[ObviousGuy]

And the government system of going with the lowest bidder is bound to cause some problems as the more expensive engineers would no doubt bring better experience and know how with them. When you bring in the inexperienced because they are cheap, you frequently end up spending more in the long run than if you had paid for the expertise up front.

It's like they say, you get what you pay for. Cheap prices are only cheap if your time has no value.

Re:All software has bugs

[Tablizer]

And the government system of going with the lowest bidder is bound to cause some problems as the more expensive engineers would no doubt bring better experience and know how with them. When you bring in the inexperienced because they are cheap...

Inexperience may be part of it, but often government systems are subject to a lot of competing interest and tying together existing diverse systems such that simple requirements in isolation often balloon into complicated situations. As a contractor your hands are often tied WRT cleaning up existing bad processes and odd requirements to solve needs of competiting agencies, departments, etc.

It is often diplomatic issues that cause the messes, not technical ones.

Re:All software has bugs

[aoteoroa]

To write perfect bug free software you must have a complete and accurate understanding of the end users problem. The best explanation I have found as to why this isn't as easy as it sounds was in a book called Software requirements and specifications which in the first chapter tells a story of a mathematician, finance director, manager, sociologist, and a stock broker discussing a recent failed project.

In 1993 the computer system project for the London Stock Exchange failed disastrously. 400 million pounds spent and nothing to show for it. Who was to pay? What had gone wrong? Why do so many developments end in disaster?

'Pure Ignorance,' said the mathematician. 'Software development is essentially a branch of mathematics. That is why computer science departments in universities have so often been closely associated with mathematics departments. You must understand that a program is a mathematical object. Its development is therefore a mathematical activity, of a particularly challenging kind. Those who engage in it should, of course, be competent both in using the appropriate mathematical notations and in drawing on the appropriate body of mathematical knowledge -that is, on knowledge of the relevant theorems. While we continue to ignore these facts we will continue to perpetrate disasters.'

'That's all very well,' said the finance director, 'but in my company we build systems to improve our business performance. I imagine that the Stock Exchange does the same. Software isn't mathematics: it's business. I think of a software development project as a capital investment. The test of its success is simply the value of the return on that investment to the company. The return in this case seems to be negative. The essential tools in a software project are financial risk analysis and discounted cash flow calculation'

'Of course you are right,' said the manager. 'But the key to achieving profitability and return on investment is to improve the development process, and with it the cost and quality of the end product. Software developers like to think they're doing something very special, but in fact it's an industrial process just like any other. The essence of software development is a quantitative approach to measuring and improving the performance of the software development process. What you don't measure you can't control.'

'But surely software development is done by people. And for people isn't it?' said the sociologist. 'Software is situated. You talk as if the system and its development were something objective, but really it has to be continually renegotiated subjectively between the various stakeholders, who all have their own agendas and perspectives. The success of any system depends directly on facilitating the negotiation, and on the determinant individual and group relationships in the societal context. I suspect that the Stock Exchange members belong to an authoritarian culture in which the dominant behaviour in inimical to peer group negotiation; perhaps that explains their failure.'

'This all seems ridiculous to me,' said the stockbroker. 'The plain fact is that the system was meant to serve the needs of brokers and jobbers of the Stock Exchange, and it didn't. It usually takes a professional working member of the exchange at least five years to learn how the Stock Exchange works, and I don't see why the analysts and programmers who make computer systems should expect to pick it up more quickly. A system for a particular business can only be built by people who are experts in that business. Domain knowledge, I think it is called. That's what matters.

Re:All software has bugs

[John Hasler]

> And the government system of going with the lowest
> bidder...

So you want them to give the contract to the most _highest_ bidder? The brother-in-law of the contract officer?

They give the contract to the lowest _qualified_ bidder. Doing otherwise would be stupid.

> It's like they say, you get what you pay for.

Bullshit.

Re:All software has bugs

[retto]

I've done some work with government organizations (not a lot so others way have had different experiences than mine) and from what I saw the biggest problem wasn't that the work was done by the lowest bidder, it was that the requirements were often created by people other than those that know the situation the best. Very little thought was given to designing something to be as usable and efficent as it can be, and more focus is given to making sure it gets finished by an arbitrary date. If it works, great, if not, it is ok as long as some department chief can say they are compliant with something by the required date. I've seen so many little problems that could have been fixed, but time involved in getting approval would have been more than then it was worth. I can't imagine the cumlative effect of all those little problems that get overlooked.

In the end you wind up with a big mess, tacking on or changing just enough to meet some kind of regulation. If you see something beyond the immediate scope of the project that would make things a lot easier and efficent, but it would require time/money or cooperation from someone else's department/division/agency it will be shot down as it won't be 'in the budget.'

Ok, that was my little rant. One time I had to sign a form to get the air conditioner turned on before the 'approved' time in a federal building, and I've been pissed off at bureaucracy ever since.

Re:All software has bugs

[Anonymous Coward]

not all windows shops and windows admins suck big giant donkey dicks.

but you DID just describe what has to be about 90% of the technical departments using MS.

Click "Next" on everything till you reach "Finish"

and know how to reboot the server.

that's about it...you are a qualified technician.

Re:All software has bugs

[homebru]

We build highways, bridges, and the like. We do the vast majority of our work under low bidder contracts. More importantly, we deliver the product on time and of a high quality.

And how do you deal with the customer whose specs say (in effect) "just throw a log across that creek because all we need is a footpath for the weekend" and subsequently declare your work an extension to the InterState Highway System and in non-compliance (substandard) of rule Blah, section Blah-blah, part Blah-blah-blah, paragraph Blah-blah-blah-blah?

From the article: The pilot was not designed to become a national system, however. The INS had intended to examine its results and then build something new, school officials who participated in the test say. It was a "throwaway project," says Johnson. "It wasn't supposed to become something bigger."

This is one of the most common causes of failure that I have seen over the years. A refusal by management to see the difference between a "proof of concept" project and a "production" project.

Attention programmers. Learn this now and learn it well. There is no such thing as a "quick and dirty" project. Anything you write for hire can and probably will be pushed into production. And if you "assumed" that you could "get by" with single user code with (for example) no record locking, error testing, logging, transactioning, or provision for remote monitoring or backup, you just screwed the pooch. The minute you check that code into CVS, it's heading for production with hundreds of incompetent users who will expect 100.000% uptime. And management will quickly point you out as the author of the failing new product and your reputation is shot, your future with the company is shot, and you have given programmers everywhere another black eye. Gee, thanks.

People, what it is, is that every piece of code that you write for hire has to be the very best you can create. Because, while your customer may have only asked you to throw a one-log footbridge across the creek, s/he is expecting an eight-lane interstate highway structure.

Re:All software has bugs

[HiThere]

The problem is probably worse than just inexperience. I don't know what it is, and I work in the middle of it. Probably it's empire-building, or at least that's my guess.

Example: I have the database that I put together for my (small agency). It took a few months to do, but not many... perhaps three, and that's allowing for a bunch of mistakes. Then it was refined for a few years.

Three years ago we were told that a State Agency would be assuming the duty. They were legally obligated to begin about a year ago. They still aren't doing it. But I built it myself, but when I talk to the State people:
1) it's a team of people building it
2) the people don't stay at the same job
3) sometimes it's being done by people in a different department.

When I went to give them my data, they wouldn't even take an electronic file. They wanted typed forms. (I cheated and printed them out on a laser printer...but the spacing had to match that of a typewritten form.)

Now this was about 4,000 sheets of paper that they were going to need to enter by hand. My *guess* is that they were entering it into a 3270 based system (i.e., old IBM mainframe) and were then going to convert it into a PC based system. They were implementing it in FoxPro...probably. I don't really know. I never ended up talking to anyone who was doing the real work.

Personally, this was what I consider my "small test system". It's the one I traditionally converted from system to system during the process of learning the new system. I'm told that they'll be on-line with it in a month. (They'd better hurry, or I'll retire before they get it running... not that it matters.)

So I sure understand about the slowness, and I have a few theories as to what causes it: You need to take the cross-product of the Parkinson Laws with the "Mythical Man-Month". Then add a bunch of Dilbert for seasoning. Gourmet crusine to revolt the finest palate.

There are times when this is a good thing

[KPU]

If the government has a harder time keeping track of people, maybe it will be less ambitious. Never mind.

Surprising?

[bajo77]

This seems to be on par with other things the government tries to keep tabs on. They can't keep track of paroled felons, the database of people who can't vote is horribly flawed, and the soundex database that the airlines use doesn't work either.

Granted, this needs to change, but this isn't the first time the government has failed to provide adequate information regarding lists of people.

Taxpayer $?

[spumoni_fettuccini]

Is the contractor just ripping off the taxpayers with bad code?

They've been doing that for years. Toilet seats for $10,000; hammers for $7,000? Not only that how much money is wasted on the old "that's exactly what I asked for, but not what I wanted"?

Re:Taxpayer $?

[Brymouse]

You forget the 10,000 toliet seat was designed for use on bombers. You can use the toliet in sever turblance, and if the plane flips, you will not get shit all over yourself. In a combat situation, this is a bargian at 10k. Imagian if your taking a shit and you come under fire, with a convetional airline toliet you would be busy cleaning up, now that could cost you your life.

Re:Taxpayer $?

[The-P]

I think something that is getting missed in this is that the government specs something, and when they spec it they don't mean buy it off the shelf (if they did they would just purchase it from a scheduled seller, not bid it out). Most of these $10,000 hammer, $7,000 toilet seat stories you hear are from NASA & aerospace bids. There is a reason these things cost ridiculous amounts of money. They are being made one at a time and they meet very stringent specifications.

P

EDS? Explains a lot...

[NickFitz]

EDS do a lot of systems that don't work, or don't work properly, or run massively over schedule and budget, here in the UK as well.

I just can't understand why governments insist on using them with the track record of cock-ups they have; they're not even cheap.

Re:EDS? Explains a lot...

[zooblethorpe]

Okay, let's engage in a little thought exercise here. It's quite likely that EDS projects <insert epithet here> because they skimp on actual project implementation. Yet the money from the contracts must be going somewhere.

Hmm. Could it be they funnel some of the excess into lobbying and other less-tasteful forms of political influence?

In the blurred lines of the public-private sector,
((Product == Crap) && (Contract == Lucrative)) ? EngagePoliticalMachinery(Future = Lots more contracts) : CutCorners(Until(Contract == Lucrative))

What, me, cynical? Nah...
Apologies for cludgy code, but then hey, I'm a translator, not a developer. :P

25 Years of Government

[Grey_Coder]

I have been working for municipalities for 25 years. I have yet to see a major program work well or work at all without overruns. I have chalked it up to me lacking a MBA or Degree in Computer Science. I am just a poor hobbiest that thinks for a million or three you should get what you pay for. But like shrinked software there must be no implied warrantee or garentee it will work. Man I think for a couple million give me a couple coders and little hardware and sit back. Open source here we come.

'Lowest Tender' syndrome.

[The Famous Druid]

I've seen this sort of thing happen before.

Government departments are pretty-much obliged to go with the lowest tender, even if the people running the tender know that the winning bidders are a bunch of incompetents who couldn't organize a fsck in a brothel.

So, the lowest bid wins, and then even if they actually are well-meaning and try to do the right thing, they have such limited resourses that they usually have to resort to working too few staff too many hours.

The result will not be quality code.

Not suprising.

[Mr. Piddle]

How hard is it to write software like this that works?

Harder than you imagine. If you remove the pork-barrel politics, directives of what technology to use coming from the clouds, and the recently potty-trained project team members, there isn't much left to give the project a chance at success. Most of the project's time is probably spent learning the difference between JDBC and EJB or at meetings discussing the differences between JDBC and EJB. The remaining time is spent accomplishing little by discussing the well-presented but vacuous system requirements for the project. Whenever I see a job posting for a database project for a government agency, I pass it and look for projects worth doing. If it mentions .NET or J2EE, I pass it by doubly fast.

I don't like this conclusion, but I've worked on, interviewed for, or heard about enough of these projects to realize they are pretty much all the same and for what seems to be all the wrong reasons.

its about "now"

[cr@ckwhore]

I have first hand experience with this subject after spending 2 long years working with a State level government agency to develop motor vehicle registration software ...

The problem is not so much about "how hard is it to write software that works" ... its more about "we're writing software for what we need RIGHT NOW".

When governments sit down to write software, its usually done through private contractors. So, a group of beaurocrats have a pow-wow and come up with a spec that generally reflects the type of work that the agency is doing "now", without much future consideration.

15 years later ... as legislation, beaurocracy, and agency regulations expand, so do the requirements of the software. For example, the Bureau of Motor Vehicles in an unspecified state put their first computer system in place in 1968. Since then, the scope of the BMV has expanded at least 10-fold.

Complicating the issue, "upgrades" are usually in the form of applying a new "layer" to the system somehow. As of 2003 in this unspecified state, the typical motor vehicle registration passes through 4 different systems before arriving in the central (OLD and limited) database at the state.

Complicating the problems even further are the many new layers of regulatory bloat -- meaning, the BMV is using software that met their needs in 1968, but doesn't meet their needs now. For example, (and this is how data goes bad), they're required to track whether or not somebody's registration is under suspension. However, back in 1968 registration suspension wasn't even a blip on the radar. To handle the problem after the "registration suspension legislation" was enacted, an "exception" had to be built into the system... if the street address field contains a special message, it indicates that the registration is under suspension. Ultimate problem... fields in the database are being used for purposes they were never intended. The age of the system does not allow for it to be updated properly.

Funny how you never hear ...

[Professor D]

The conspiracy theorists talk about how damned inefficient, bloated, clumsy and self-defeating government agencies projects are.

Somehow "they" have had UFO technology which would make petroleum obsolete since the '50s, conspired to kill JFK to keep it a secret, brainwashed Chapman to murder Lennon, created a secret government database tracking everyone's cash transactions, control us by putting chemicals in our water and thought patterns in satellite broadcasts. Oh yeah and "they" also were behind the 9/11 attacks as well.

Yet "they" can't even figure out how to keep track of whether or not foreign students went to class or not.

MS BS

[coyote-san]

That's MS BS. (And the cry of incompetent programmers for decades.) Even if we agree that all software has bugs - and I don't - that canard says nothing about all bugs being equal much less anything about all software having about the same number of bugs.

Any competent manager would know that experienced coders are usually FAR cheaper than inexperienced ones because they make fewer mistakes due to ignorance or indifference ("it works for me, so it's done!"). That gets you to the point of dealing with the more subtle and intrinsic bugs (e.g., due to conflicting requirements) quicker and cheaper, and the apparent cheaper cost of inexperienced developers is only achievable if you plan to release after coding is finished, not after testing is completed. Which is pretty much every MS *.0 release, now that I think about it -- got to get to market first, even if it's pure crap!

Sounds like poor relational design

[GFW]

Unfortunately, no technical details were given (such as programming language(s), database system, and general architecture type. However, a number of the bugs (like the one about not being able to graduate from a BA to Masters, and the one about the birth of a child) suggest an underlying poor relational design. If I knew more about the overall architecture, I could comment more about how many bugs one would expect. Certainly, other people have commented that you can have lots of bugs *before* you release and start taking the program seriously... I understand EDS has often gotten away with shoddy work for the government. I don't know why (political payoffs).

And another thing...

[Mattcelt]

I think there's an even darker side than what's being presented here in the brief - consider what happens when one of these 'glitches', whether techinical or PEBKAK, cause inaccurate information to be propogated through the linked government databases such as the TIA? Does joe traveller get strip-searched at every airport he goes to because someone "accidentally" put his name onto a terrorist watch list? Where does the government's responsibility to be accurate and precise with our information end? If the Credit Reporting Agencies are any indication, I think we have a potentially huge problem on our hands.

My guess...

[adagioforstrings]

If it's anything like my work, it probably started out as an Access (etc) database created by someone with limited knowledge of database design to try to make their job easier. It worked well enough and someone decided to get a contractor to flesh it out (perhaps explaining why the contractor referred questions back to the govnt: they are limited by using existing database or something). I don't think the contractor is necessarily ripping off the taxpayers, but who knows. Still, you'd think they'd do a better job of it. I think it's something started small, but got too big too fast. Happens all the time in companies, but has bigger impact when it happens in the government.

System doesnt work

[t0ny]

I work with the government as a consultant, and I continue to be amazed at the waste and stupid use of money involved in government projects. What seems to be the rule, rather than the exception, is poorly skilled people pitching initiatives they have absolutely no skills to impliment, and they dont even put in the work to design it well.

There is one database used for payroll on which millions (if not tens of millions) were spent, and the end result is a system nobody is completely sure of, and which requires all the deparments involved to completely change all their procedures. And its even less flexible and problematic than what it is replacing. AND this is a custom application!

Until government starts paying tech people what they are worth in the private sector, you will always have poorly skilled bullshitters pedalling their wares to the public sector, who is suffering from the illusion that throwing money at a problem will make it go away.

Legacy systems

[shogarth]

I can see how the individual campuses could be in the hurt locker. For obvious reasons, they are not going to want to do manual updates every time an address changes. That means integration with existing systems that (hopefully) allow students/faculty/staff to update their own records.

Many campuses (at least state campuses) are running really old student records systems. Imagine that the web front end for the users is built from screen-scrapes of old mainframe applications. To make it worse, now try to interface 20+-year-old, non-relational, pre-SQL student records systems with a new distributed database model.

The result is that the schools will have to reallocate senior programmer (who else will know how the records system works) time bumping other, mission-oriented tasks in the name of National Security.

Re:Anyone seen "Brazil"?

[qtp]

what's frightening about Ashcroft is not that he's a fundamentalist autocrat, but that he's an incompetent fool.

The real danger of John Ashcroft is that he's a fundamentalist autocrat and an incompetent fool.

Bureaurcy and the lowest bidder

[bastion]

As a matter of public policy (and taxpayer money) is this level of non-functionality to be expected in these sorts of projects?

Yes.

Look on the bright side, if they can screw up simple projects like this how far do you really think TIA (Total Infomation Awareness) is gonna get?

TIA Alias Search: Commander Taco

Output: Mexican terrorist. Leader of collbration site for social dissidents.

Just far enough from the truth to get somebody sued....

Re:Wonder what the SEVIS site is running?

[pinko-rat-bastard]

I was wondering that myself, especially after reading this quote from the article:

The association notes an anonymous report of technicians telling one school that a "catastrophic failure" had occurred while processing its records. When school officials asked what had caused the error, and what "catastrophic" meant, the technicians said they didn't know.

It may just be a coincidence, but this is (literally) an error message sometimes reported by ADO (Microsoft's Active Data Objects)

All software has bugs

[Tony-A]

The thing is to be aware of this before coding and take steps to compensate and reduce the damage from those bugs.
An excuse after the fact, the canard seems infantile.

All software has bugs. They are most certainly not created equal. Just because there is no way to expose them, doesn't mean they're not there.

("it works for me, so it's done!")
OUCH!
There is a big difference between partially working for a few things for a few people and never failing for everything for everybody. Some of the critical knowledge lies in the application area and is neither achieved readily or even necessarily consistent.

Brief Rebuttal

[Bios_Hakr]

I have been working in the USAF for about 8 years. 6 of that in WAN (longhaul voice and data), and 2 in Infrastructure and security. I'd like to offer another side to your story:

>but it needed to be done before the end of the fiscal year

This is how it works: The USAF has a budget. Each area gets a small slice. If filters down to each office having about $10k ~ $30k for operations that year. That money has to last all year. About 20% of that is kept in reserve funds. If that money is not needed by August, we are free to spend it. At that point, we develop a wish list and try to get that aproved. By time all this happens, we have about 5 weeks to spend the reserve money.

No one in the military likes it. All our contractors hate it. If you want it changed, write your congressperson and have them change 50+ years of bad management practices...

>The quality of code would've been greately improved if we coded, say 40 hrs/week instead of pulling all-nighters.

I have spent countless days and nights working overtime. So have a lot of my coworkers. In times of exercise or, God forbid, a war, we go to 12+ hour days. 6 days on and 1 day off are common during exercises.

Contractors always make fun of us for sloppy wiring, half-assed installs, unpatched servers, etc... When new equipment arrive, we usually have a few hours to determine where it will go and when. We are usually told that the old equipment stays in place until the new stuff is operational. This leads to massive misuse of rack space. and cluttered wiring.

Also, just like your code suffers from 40+ hours, my wiring suffers when I have to spend my Saturday morning connecting a new router.

No one likes to work overtime. Your work suffers just like mine. You may lose a contract because of your bad code. People could lose lives because of my bad wiring. Let's both work harder to keep our shit straight, regardless of hours worked.

>They insisted on .NET 2003 server with M$ SQL, etc., etc.

This is becuase we have a very nice license with MS for their stuff. We get good support, including semi-annual "Best Practices" reviews by MS inspectors. The US Gov paid for MS tools, we should use them. If you don't like it, write your congressperson. Personally, I'd love to be able to use Squid on Red Hat. Unfortunately, we don't have the money to spend on more software licenses after we bought MS stuff.

>asked if it would work with a win2003 server as opposed to a win2k

Our upgrade paths are fixed by MS. This absolutely sucks. Our systems require specific patch releases from MS. Once they stop supporting those patch paths, we have to upgrade. Agian, if you don't like it, write your congresscritter.

>but they didn't even know how to install windows

I'm throwing a bullshit flag on this play. I find it difficult to belive that no one knew how to install Windows. In the USAF, we have a NCC department that does nothing but install, configure, and maintain Win2k servers.

There may have been an internal power play based on getting Win2k3 server training. That is an ongoing military issue. Your boss tells you to do something. If you do it and screw it up, they ask if you were trained to do that thing. If you were not trained, then you go to federal-pound-me-in-the-ass prison for working on something without proper training. If you were trained and you screw it up, then you get in trouble for not folowing the training guidelines for whatever it was you broke.

Everyone working in a military NCC can install Win2k Workstation and Server. Many of them are MSCEs or higher. They could probably install Win2k3. They just wanted official training on that product before they tried something and broke it.

>Installing a a windows server is a mountain of a task for them.

No it isn't.

>Installing .NET is something that, as they say, they 'have been working on for

It's 'Unit Tests', stupid!

[adamscottphotos]

I'm amazed that I haven't seen a single poster bring this up yet so...

The key to quality software; flexible, extensible, fault-tolerant, maintainable, and all of those other adjectives that 'good' software is supposed to have is very, very simple.
It's called Unit Testing. It's not brain surgery. I have worked on several medium to large scale projects (500k-3.5m lines) in several languages and environments, and I've yet to bomb one hard using this methodology, despite the usual client shenanigans.

Every time I write a functional subunit, I start by writing a series of tests (based on the spec, hopefully) that define 'doneness' for that subunit. Every object in the system has it's own set of tests. The test harnesses are chained together, so I can hit a button, so to speak, and run all of the hundreds to thousands of tests at once.

Whenever I check in new code changes, I run the test suite. If a test fails that previously worked, then I broke something. This plus good OOP practices (low coupling, high cohesion) allows you to make changes on the fly without the kind of 'The Money Pit' syndrome (fix one thing, another breaks) that is described in the article.

I am certain that the system in question was NOT developed with these methods. Most development organizations that I come into contact with pay lip service to the concept, but don't want to spent the perceived extra $ up front. The thought of all those developers writing TESTS when they could be writing CODE scares the willies out of them. But it pays for itself. It really does, every single time. Don't tell your boss, and try it on your next project. This is old news - google has a ton of info on it, and there are some good but unnecessary books also.

In the dot.com glory days, we had a huge system, running several hundred transasctions per second on a geographically distributed system of clients. We made fundamental architectural changes without a hitch, switched servers live without a hitch etc, and made a zillion little changes, all live, and all without a hitch (well, other than really stupid human errors, like locking out the client upgrade system with a bad password... oops). We had zero budget, and 2.5 developers. Unit tests are the Way, and if any company that doesn't mention them in your first meeting, run like hell.

Reactive, Just-In-Time Development

[prestidigital]

I've read article and many of the replies here. Several /.ers definitely describe flaws in government contracting processes and hiring practices that I've experienced, but I think they are missing the point. I think there is an additional, more fundamental flaw, that has been overlooked - or at least didn't get modded up high enough for me to see - maybe i should go trolling for a different set of opinions :^).

My experience tells me that the problems begin when we fall into the trap of trying to solve problems with a reactive mindset instead of a proactive mindset (proactive being favorable). We allow daunting problems and/or a need for revenue to back us into a corner time and time again and every time we are forced to hack our way out. Some of that is just old-fashioned survival, but a lot of it can be avoided with deliberate forethought, planning, discipline, and a commitment to quality and detail.

Avoiding clusterf*x has to be an institutional effort, whether the institution is a huge goverment agency or an tee-tiny, independent software shop. Everyone in the organization - operations, sales, IT - has to be on board with the policy that "if it's worth doing at all, then it's worth taking the time to do it right...the first time." I said "fundamental" earlier b/c that has to be something like lesson #5 in life's little handbook - we all heard it all too often when we were kids, we know it's true, and yet now we don't pay it any mind.

I do think the failure to heed that simple maxim usually starts in business development and snowballs by the time it gets to IT, but it really goes both ways. Everyone has to be responsible for maintaining the discipline required to produce quality.

What happened with this system is everyone involved got themselves all in a panic like a drowners who not only won't let you save them, but pull you under too. It's understandable given Sept 11, but "undertandable" and "right" are two different things. Legislators threw money at a situation they didn't try to understand. Deal-makers when after that money, promising to solve problems they knew they didn't understand. Developers enabled deal-makers by claiming to understand. No one took the time to do it right the first time.

P.S. It doesn't have to be this way.

Look at is this way...

[tundog]

Maybe the bugs where put there by a malicious programmer (slashdotter?) who disagrees with all this tracking nonesense. No better way to sow the seeds of discontent.

I was talking about something similar to a colleague about how I wouldn't be a part of a research proposal for the DoD because I fundamentally disagree with concept of a Big Brother State. He said that he actually new a researcher who had taken on defense-funded research projects in the past and just tanked-it on deliverables because of his own ideological sentiment. Not only was the research rendered innocuous, but it tied up government funds that could have been used on other more incideous projects.

Not an approach I would champion, but interesting nonetheless...

Don't worry, screw-ups are not just for government

[NotQuiteReal]

Having worked for some large companies (directly or contractually) I have seen plenty of IT disasters in private industry.

The common thread is typically not an engineering failure, but rather, egos, politics, and ill defined requirements.

Like many things in life, there is usually no big conspiracy to screw things up, they just get that way "one good idea" at a time (e.g. The road to hell is paved with good intentions).

Too much job security?

[BenjyD]

The problem with most of these projects seems to be the lack of that standard incentive scheme most companies run which goes:

Do job really badly -> lose job

Here in the UK, government IT projects go vastly over budget and fall apart time after time, yet the same companies get hired again and again to do the contracting.

Re:programmers =~ lawyers

[ctve]

If I had some points, I'd mod it.

It's quite interesting to me, as I've been thinking about what attributes I got from my parents. My father is a lawyer, and I became a programmer, and I also have an amateur interest in the law.

Both are based on rules, and both are based upon a certain degree of exactitude. One thing that I notice that many people I know don't is when a law is badly drafted and ambiguous, as I think it comes from the part of my brain that sees badly written and ambiguous specifications.

You are the government

[Anonymous Coward]

I work in IT for a municipal govt.

The problem with government is that you don't have a unifying purpose such as say a profit motive. Each government agency is like a separate business with different agendas. Getting them to work together for a common good (like say the tax payers) is very difficult.

One of the biggest problems in government is the notion that you can't get rid of anyone. We have no policies that say you cannot get rid of dead wood, just an organizational lack of intestinal fortitude to do just that.

The other big problem is that many govt workers are just hanging around until they can get a state or federal retirement. Great for them but not so great for the organization or the citizens.

If you are lucky you will find some very good IT people in govt. If you are unlucky you will find zombies waiting around for for retirement with a scope of reference that revolves around the last time they actually could do anything (usually back in the 1980s or early 1990s). If you are really unlucky your PHB will get all of their information from industry fish wraps or from Gartner reports...you wouldn't want to say think for yourself or get informed...

The funny thing is that govt exists to serve the citizens. I almost never hear govt employees talk about the citizens or look at what may be in the citizens best interests. Doing things better and cheaper seems to be in the citizens' best interests. Chasing technology for the sake of technology doesn't seem to be. Working together across the organization seems to be in the citizens' best interest. Building empires and conducting turf wars to protect and expand the empire is not.

EDS not Technology failure

[xdonnel]

Cost is rarely the only criteria on large public sector contracts
With EDS's track record of failure and their financial instability,
it is hard to understand why they still receive these contracts.

This application could be developed successfully on any modern platform

EDS is most likely responsible for most issues raised in the article if they were the main contractor.

The "bleeding" issue is the most serious issue raised.
If the system cannot maintain data integrity, then it is worthless for
the main requirement of monitoring students who may be a security threat.

The "bleeding" issue is probably caused by
failure in database design (EDS resp)
or less likely a serious performance issue

"to create a record for the newborn child.. ..delete everyone's file and create new ones"
failure in database design (EDS resp)

"Before extra capacity was added to the system, several student advisers
logging on at once caused SEVIS to slow dramatically"
Flawed operational architecture even if sizing, availability requirements were poorly defined as solution did not scale (EDS resp)

"help desk for long periods, only to be offered a solution that turned out to be wrong"
Failure possibly due to organization, staffing level, poor documentation, inadequate handover to maintaining organization to provide support at the required SLA (EDS resp)

"..several others, she says. "It begs the question,
how rickety is [SEVIS] if you can't do upgrades to it?"
poor maintenance procedures, no regression testing (EDS resp if maintaining)

"The Bureau of Immigration and Customs Enforcement
has yet to name an official to oversee SEVIS."
Inadequate handover to maintenance organization (joint failure)