Glitches in Massive Government Databases?

HBergeron asks: "Rather then post this as another YRO in the litany of new government datamarts there is a more fundamental question for all the coding Slashdot readers out there. This story, in Government Executive magazine, outlines the range of programming glitches in what is a relatively simple database. As a matter of public policy (and taxpayer money) is this level of non-functionality to be expected in these sorts of projects? Is the contractor just ripping off the taxpayers with bad code? How hard is it to write software like this that works?" The article focuses on the SEVIS database, but have others noticed similar trend in other government information systems?

Comments

All software has bugs

[ObviousGuy]

And the government system of going with the lowest bidder is bound to cause some problems as the more expensive engineers would no doubt bring better experience and know how with them. When you bring in the inexperienced because they are cheap, you frequently end up spending more in the long run than if you had paid for the expertise up front.

It's like they say, you get what you pay for. Cheap prices are only cheap if your time has no value.

Re:All software has bugs

[Xerithane]

It's like they say, you get what you pay for. Cheap prices are only cheap if your time has no value.

Wait! But what about Linux?

Time to end the sarcasm for the day..

Re:All software has bugs

[Tablizer]

And the government system of going with the lowest bidder is bound to cause some problems as the more expensive engineers would no doubt bring better experience and know how with them. When you bring in the inexperienced because they are cheap...

Inexperience may be part of it, but often government systems are subject to a lot of competing interest and tying together existing diverse systems such that simple requirements in isolation often balloon into complicated situations. As a contractor your hands are often tied WRT cleaning up existing bad processes and odd requirements to solve needs of competiting agencies, departments, etc.

It is often diplomatic issues that cause the messes, not technical ones.

Re:All software has bugs

[aoteoroa]

To write perfect bug free software you must have a complete and accurate understanding of the end users problem. The best explanation I have found as to why this isn't as easy as it sounds was in a book called Software requirements and specifications which in the first chapter tells a story of a mathematician, finance director, manager, sociologist, and a stock broker discussing a recent failed project.

In 1993 the computer system project for the London Stock Exchange failed disastrously. 400 million pounds spent and nothing to show for it. Who was to pay? What had gone wrong? Why do so many developments end in disaster?

'Pure Ignorance,' said the mathematician. 'Software development is essentially a branch of mathematics. That is why computer science departments in universities have so often been closely associated with mathematics departments. You must understand that a program is a mathematical object. Its development is therefore a mathematical activity, of a particularly challenging kind. Those who engage in it should, of course, be competent both in using the appropriate mathematical notations and in drawing on the appropriate body of mathematical knowledge -that is, on knowledge of the relevant theorems. While we continue to ignore these facts we will continue to perpetrate disasters.'

'That's all very well,' said the finance director, 'but in my company we build systems to improve our business performance. I imagine that the Stock Exchange does the same. Software isn't mathematics: it's business. I think of a software development project as a capital investment. The test of its success is simply the value of the return on that investment to the company. The return in this case seems to be negative. The essential tools in a software project are financial risk analysis and discounted cash flow calculation'

'Of course you are right,' said the manager. 'But the key to achieving profitability and return on investment is to improve the development process, and with it the cost and quality of the end product. Software developers like to think they're doing something very special, but in fact it's an industrial process just like any other. The essence of software development is a quantitative approach to measuring and improving the performance of the software development process. What you don't measure you can't control.'

'But surely software development is done by people. And for people isn't it?' said the sociologist. 'Software is situated. You talk as if the system and its development were something objective, but really it has to be continually renegotiated subjectively between the various stakeholders, who all have their own agendas and perspectives. The success of any system depends directly on facilitating the negotiation, and on the determinant individual and group relationships in the societal context. I suspect that the Stock Exchange members belong to an authoritarian culture in which the dominant behaviour in inimical to peer group negotiation; perhaps that explains their failure.'

'This all seems ridiculous to me,' said the stockbroker. 'The plain fact is that the system was meant to serve the needs of brokers and jobbers of the Stock Exchange, and it didn't. It usually takes a professional working member of the exchange at least five years to learn how the Stock Exchange works, and I don't see why the analysts and programmers who make computer systems should expect to pick it up more quickly. A system for a particular business can only be built by people who are experts in that business. Domain knowledge, I think it is called. That's what matters.

Re:All software has bugs

[vladkrupin]

The quote from the original article:
How hard is it to write software like this that works?

Wow! Well said! My grandma couldn't have done better. In other words, please define 'works' for me. How many blue screens a day constitute 'works' and how many are too many?

Also, since we are at it, I want to reflect back on the latest project we have done. Incidentally, for the government. Before asking if a vendor is ripping the taxpayer off we need to consider how the government mismanages the resources it has. Consider the facts:

1. The project itself was fairly small and simple. I'd say it would normally take about 2 months to develop and deploy, but it needed to be done before the end of the fiscal year, so it was a 'now-or-never' situation, and was a horrible time-crunch. We had slightly more than half the time necessary to do it, but they won't even try to install it till probably the end of the year! The quality of code would've been greately improved if we coded, say 40 hrs/week instead of pulling all-nighters.

2. They tried to keep tabs on the development by scheduling 'technical meetings' over the phone. While there is nothing wrong with that per se, in a time-crunch that was a horrible waste of time. The smartest things we've heard from them were questions like 'Are you using hungarian notation?' or 'is your code well-documented?'....

3. They insisted on .NET 2003 server with M$ SQL, etc., etc. We did our best to make them consider PHP and the like, because that's what we normally use, but they were willing to pay extra to have that stuff developed in all-M$ stuff! We were told that the reason for that was because their IT was managing only M$ software, and the server was already there, and they couldn't have anything else (e.g. PHP). Fine, I can understand some bureaucracy in IT - that's cool, but imagine my surprise when, after we shipped them a CD with the project, they called us back and asked if it would work with a win2003 server as opposed to a win2k!!! Not only they didn't have the server yet, (or the infrastructure for that matter), but they didn't even know how to install windows! Which brings me to point #4:

4. Their IT is kick-ass. As in 'their ass needs to be kicked real hard'. Installing a a windows server is a mountain of a task for them. Installing .NET is something that, as they say, they 'have been working on for a while, but haven't got it quite yet'! And, when we give them a database dump they have no idea what to do with it and you have to walk them through the process (right-click on the 'Databases', select 'Create New database', click ok...) And they are paying these people!!! Errr... Let me re-phrase that - We are paying the government to employ those dumbheads! Thatnks goodness the network on which that is installed is not connected to the internet - the same idiots are in charge of security as well.

Yes, it is true that some contractors will rip off the government (and it is really the government's responsibility to make sure that doesn't happen! But that's not the point). The point is that even if they have a perfectly good product developed by honest people, they are still remarkably talented at screwing it up. Bureaucracy and lots of idiots in charge of hiring people are to blame.

Re:All software has bugs

[hendridm]

> And the government system of going with the lowest bidder is bound to cause some problems

I worked in a state agency, and the fact that we were required to take bids didn't really change who we purchased from. We just chose the vendor we liked best and justified it by writing the project needs around that vendor. They did that with employees too. When a new job opened, they often had someone in line for the position. However, equal opportunity required that the do interviews for the position. To justify the person they desired, they would write the job description and requirements based on the skills of the individual they liked. They would then schedule interviews even though they already had someone chosen for the position, just to meet requirements. I suppose they could have changed their mind if they found someone who was absolutely fabulous, but it's hard to convince an employer how great you are when in the back of their mind they don't think the interview is going to matter anyway.

Re:All software has bugs

[John Hasler]

> And the government system of going with the lowest
> bidder...

So you want them to give the contract to the most _highest_ bidder? The brother-in-law of the contract officer?

They give the contract to the lowest _qualified_ bidder. Doing otherwise would be stupid.

> It's like they say, you get what you pay for.

Bullshit.

Re:All software has bugs

[retto]

I've done some work with government organizations (not a lot so others way have had different experiences than mine) and from what I saw the biggest problem wasn't that the work was done by the lowest bidder, it was that the requirements were often created by people other than those that know the situation the best. Very little thought was given to designing something to be as usable and efficent as it can be, and more focus is given to making sure it gets finished by an arbitrary date. If it works, great, if not, it is ok as long as some department chief can say they are compliant with something by the required date. I've seen so many little problems that could have been fixed, but time involved in getting approval would have been more than then it was worth. I can't imagine the cumlative effect of all those little problems that get overlooked.

In the end you wind up with a big mess, tacking on or changing just enough to meet some kind of regulation. If you see something beyond the immediate scope of the project that would make things a lot easier and efficent, but it would require time/money or cooperation from someone else's department/division/agency it will be shot down as it won't be 'in the budget.'

Ok, that was my little rant. One time I had to sign a form to get the air conditioner turned on before the 'approved' time in a federal building, and I've been pissed off at bureaucracy ever since.

Re:All software has bugs

[Mr. Piddle]

Domain knowledge, I think it is called. That's what matters.

They are all right, except for the mathematician, of course (just kidding...well, mostly).

My own personal list of priorities would be (from high to non-existant):

1) Domain knowledge.
2) Solid data modeling ability.
3) Most everything else.
1,000,000) Buzzword compliance and/or magazine-cover conformance.

Unfortunately, most self-proclaimed prophets of the software world work in the exact opposite order, which is why nearly every project is a pain in the ass.

Re:All software has bugs

[big tex]

..As I step on my soapbox, donning the asbestos...

This, written by and modded "+5, Insightful" from the same group of programers that:

1) bitched about the NSPE coming down on CS's (emphasis on the science part of CS) calling themselves Engineers with a capital letter;
2) wonders why the IT market is softer than a sponge;
3) Actually debates the need for college education in CS;

and so on.

Now, I'm sure that there are quite a few excelent CS people who deserve the moniker "Engineer" (Linus comes to mind, occuping the other end of the spectrum from RMS).

As for the lowest bidder system, I work for a construction contractor. We build highways, bridges, and the like. We do the vast majority of our work under low bidder contracts. More importantly, we deliver the product on time and of a high quality.
How and why, you ask? Quite simply, because we must. We must because we said that we would. We must because the state deserves a fine highway for the money they give us. We must so that we can stay in business and make more money tomorrow. We must so that the system keeps working.

If you can't do what you do correctly, give it up. You are the people Ayn Rand warned us about.

Re:All software has bugs

[homebru]

We build highways, bridges, and the like. We do the vast majority of our work under low bidder contracts. More importantly, we deliver the product on time and of a high quality.

And how do you deal with the customer whose specs say (in effect) "just throw a log across that creek because all we need is a footpath for the weekend" and subsequently declare your work an extension to the InterState Highway System and in non-compliance (substandard) of rule Blah, section Blah-blah, part Blah-blah-blah, paragraph Blah-blah-blah-blah?

From the article: The pilot was not designed to become a national system, however. The INS had intended to examine its results and then build something new, school officials who participated in the test say. It was a "throwaway project," says Johnson. "It wasn't supposed to become something bigger."

This is one of the most common causes of failure that I have seen over the years. A refusal by management to see the difference between a "proof of concept" project and a "production" project.

Attention programmers. Learn this now and learn it well. There is no such thing as a "quick and dirty" project. Anything you write for hire can and probably will be pushed into production. And if you "assumed" that you could "get by" with single user code with (for example) no record locking, error testing, logging, transactioning, or provision for remote monitoring or backup, you just screwed the pooch. The minute you check that code into CVS, it's heading for production with hundreds of incompetent users who will expect 100.000% uptime. And management will quickly point you out as the author of the failing new product and your reputation is shot, your future with the company is shot, and you have given programmers everywhere another black eye. Gee, thanks.

People, what it is, is that every piece of code that you write for hire has to be the very best you can create. Because, while your customer may have only asked you to throw a one-log footbridge across the creek, s/he is expecting an eight-lane interstate highway structure.

Re:All software has bugs

[HiThere]

The problem is probably worse than just inexperience. I don't know what it is, and I work in the middle of it. Probably it's empire-building, or at least that's my guess.

Example: I have the database that I put together for my (small agency). It took a few months to do, but not many... perhaps three, and that's allowing for a bunch of mistakes. Then it was refined for a few years.

Three years ago we were told that a State Agency would be assuming the duty. They were legally obligated to begin about a year ago. They still aren't doing it. But I built it myself, but when I talk to the State people:
1) it's a team of people building it
2) the people don't stay at the same job
3) sometimes it's being done by people in a different department.

When I went to give them my data, they wouldn't even take an electronic file. They wanted typed forms. (I cheated and printed them out on a laser printer...but the spacing had to match that of a typewritten form.)

Now this was about 4,000 sheets of paper that they were going to need to enter by hand. My *guess* is that they were entering it into a 3270 based system (i.e., old IBM mainframe) and were then going to convert it into a PC based system. They were implementing it in FoxPro...probably. I don't really know. I never ended up talking to anyone who was doing the real work.

Personally, this was what I consider my "small test system". It's the one I traditionally converted from system to system during the process of learning the new system. I'm told that they'll be on-line with it in a month. (They'd better hurry, or I'll retire before they get it running... not that it matters.)

So I sure understand about the slowness, and I have a few theories as to what causes it: You need to take the cross-product of the Parkinson Laws with the "Mythical Man-Month". Then add a bunch of Dilbert for seasoning. Gourmet crusine to revolt the finest palate.

There are times when this is a good thing

[KPU]

If the government has a harder time keeping track of people, maybe it will be less ambitious. Never mind.

Surprising?

[bajo77]

This seems to be on par with other things the government tries to keep tabs on. They can't keep track of paroled felons, the database of people who can't vote is horribly flawed, and the soundex database that the airlines use doesn't work either.

Granted, this needs to change, but this isn't the first time the government has failed to provide adequate information regarding lists of people.

Re:Surprising?

[Guppy06]

"They can't keep track of paroled felons, the database of people who can't vote is horribly flawed,"

Then don't live in Florida. It was a system designed to try to help Florida local governments enforce a Florida state law. This has absolutely nothing to do with the federal government, which is what the federal constitution mandates.

The way you genericly complain about "the government" shows one of the problems in these information systems: a total lack of knowledge of how "the government" works here in the US. We don't have one monolithic government (yet), we have a federal government, 56 state/commonwealth/district/territorial governments, and 3,066 county/city/parish/borough governments. Each of them have different rights and responsibilities spelled out by constitutions and charters. Each of them operates in a slightly different manner from all the others to reflect the wants and needs of its people.

I'm reminded of a story here on Slashdot a few years back about how France's government went "online," giving internet users the ability to do things like get a driver's license or enroll in public schools on a website. There were a great many people who complained about how the US hasn't done anything like this, all by people who didn't seem to realize that just about everything this new French site did are the responsibilities of state and local governments here in the US.

The only way things like this will get easier is if the US shifts away from the federal republic model and become a monolithic republic ala France. This would involve disemboweling the federal constitution and burning all state constitutions. Personally, I'd rather not see that happen.

Taxpayer $?

[spumoni_fettuccini]

Is the contractor just ripping off the taxpayers with bad code?

They've been doing that for years. Toilet seats for $10,000; hammers for $7,000? Not only that how much money is wasted on the old "that's exactly what I asked for, but not what I wanted"?

Re:Taxpayer $?

[Brymouse]

You forget the 10,000 toliet seat was designed for use on bombers. You can use the toliet in sever turblance, and if the plane flips, you will not get shit all over yourself. In a combat situation, this is a bargian at 10k. Imagian if your taking a shit and you come under fire, with a convetional airline toliet you would be busy cleaning up, now that could cost you your life.

Re:Taxpayer $?

[spumoni_fettuccini]

The reason the government pays $10,000 for toilet seats has less to do with being bad at finding low prices

I spent time in Uncle Sam's Misguided Children, so yeah I knew there is/was fudging. Also [personal experience and no I was not in S4, logistics and supply] if a unit/department did not spend its entire allocated budget by the end of the fiscal year said budget was reduced. There were massive buying sprees of barracks supplies, spare parts, office furniture, etc. While not wasteful, unless it had a short shelf life, we wound up buying all kinds of crap that sat around for months before it was used. I'm not disagreeing with you; I'm just adding another expense theory.

Re:Taxpayer $?

[RALE007]

The government "spending" $10,000 per toilet seat, or $7,000 per hammer, is not due to some cunning shady "contractor" (vendor would be a better word) having duped poor innocent unsuspecting Uncle Sam into paying ridiculously inflated prices. The real reason(s) can be attributed to one of three likely candidates in my opinion:

1) A government employee(s) received a very large kickback from contractor (vendor) by accepting to pay the outrageously inflated prices.

2) The accounting department doesn't want their budget cut, so they happily pay inflated prices to "prove" they will need the same or greater funds next year.

(On a side note, the second reason happens all too often. I used to set up enterprise solutions for HP, eg tell IT departments what they need to buy from us to be able to do what they want. Half the time you talked to a government office, they had no project or system to set up, they just wanted to spend money to keep their budget.)

3) Government office accountant tacks on $6,995 to the actual $5 spent on a hammer to hide the funds he's been embezzling.

The situation is ridiculous, but you don't give the government enough credit for being a part of the problem. They're not only stupid, they're a bunch of dirty thieves too. Public servants my lily white....

Re:Taxpayer $?

[jratcliffe]

Not this again. Once again, the reason for the $500 hammers, etc., is a matter of cost accounting, not nefarious fraud. These "scandals" erupted because the cost accounting used for a number of government contracts spread the large overhead and R&D costs equally by product line item. In other words, if there were 1,000,000 items being purchased, and $500,000,000 in R&D costs, each item was assigned $500 in R&D cost, whether it was a $50,000 custom circuit board or a $10 hammer. A lousy way to do cost allocation? You betcha. Fraud? Nope. See this for more info.

Government Waste

[simsj]

This make me glad I don't pay taxes

Had to.

[Valar]

Neo: I just had a deja vu.
Morpheus: What? What did you see?
Neo: I saw the same Bush pass by twice.
Morpheus: Was it exactly the same Bush?
Neo: I dunno... could've been some kind of father son thing.
Morpheus: A deja vu is a glitch in the database. It usually happens when they change something. Particularly, votes.

EDS? Explains a lot...

[NickFitz]

EDS do a lot of systems that don't work, or don't work properly, or run massively over schedule and budget, here in the UK as well.

I just can't understand why governments insist on using them with the track record of cock-ups they have; they're not even cheap.

Re:EDS? Explains a lot...

[Onetus]

Actually they do a lot of systems that work. And work so well - they're never mentioned because there's no publicity when they just do their job. Since they're the founder of the IT Services industy and currently #2 in IT Services (after I.B.M.) they have done a lot of projects - worldwide.

If you want to believe everything the media reports and not read between the lines, go ahead. Just don't try and peddle it to the rest of us.

25 Years of Government

[Grey_Coder]

I have been working for municipalities for 25 years. I have yet to see a major program work well or work at all without overruns. I have chalked it up to me lacking a MBA or Degree in Computer Science. I am just a poor hobbiest that thinks for a million or three you should get what you pay for. But like shrinked software there must be no implied warrantee or garentee it will work. Man I think for a couple million give me a couple coders and little hardware and sit back. Open source here we come.

This isn't really news

[dirtfirst]

...Except perhaps to the executives the magazine is aimed at. Early versions of software are generally pretty buggy, particularly if the target keeps changing, and most especially if it is in response to a hastily crafted law. The only thing that's surprising about this is that the output is taken so seriously by law enforcement officials *prior to completion*.

Don't they have some donuts that need eating?

'Lowest Tender' syndrome.

[The Famous Druid]

I've seen this sort of thing happen before.

Government departments are pretty-much obliged to go with the lowest tender, even if the people running the tender know that the winning bidders are a bunch of incompetents who couldn't organize a fsck in a brothel.

So, the lowest bid wins, and then even if they actually are well-meaning and try to do the right thing, they have such limited resourses that they usually have to resort to working too few staff too many hours.

The result will not be quality code.

What is this a surprise?

[mc6809e]

When it comes to government, failure is rewarded with more money. In fact, failure is often cited as proof not enough money is being spent.

Besides, why bother doing a good job if you know you'll get paid either way? That's what the tax collector is there for!

Anyone seen "Brazil"?

[the gnat]

I think this movie shows what a *real* totalitarian state would look like: the danger to our freedoms is not from corruption but from incompetence. Programs like TIA creep me out because I'm absolutely certain that the Feds will find a way to fuck it up and throw some poor guy in detention because the computer skipped a byte and came up with his name. Ditto for the PATRIOT Act. Few people have recognized this, but what's frightening about Ashcroft is not that he's a fundamentalist autocrat, but that he's an incompetent fool. If innocent people suffer from the government's extension of powers, it won't be due to the GOP taking out its enemies but because some FBI secretary got a virus on her computer.

I'm not a libertarian; the government indirectly pays a large portion of my salary. However, the extension of government power worries me, because the more control they have, the more opportunities to fuck our lives up.

Re:Anyone seen "Brazil"?

[qtp]

what's frightening about Ashcroft is not that he's a fundamentalist autocrat, but that he's an incompetent fool.

The real danger of John Ashcroft is that he's a fundamentalist autocrat and an incompetent fool.

Not suprising.

[Mr. Piddle]

How hard is it to write software like this that works?

Harder than you imagine. If you remove the pork-barrel politics, directives of what technology to use coming from the clouds, and the recently potty-trained project team members, there isn't much left to give the project a chance at success. Most of the project's time is probably spent learning the difference between JDBC and EJB or at meetings discussing the differences between JDBC and EJB. The remaining time is spent accomplishing little by discussing the well-presented but vacuous system requirements for the project. Whenever I see a job posting for a database project for a government agency, I pass it and look for projects worth doing. If it mentions .NET or J2EE, I pass it by doubly fast.

I don't like this conclusion, but I've worked on, interviewed for, or heard about enough of these projects to realize they are pretty much all the same and for what seems to be all the wrong reasons.

its about "now"

[cr@ckwhore]

I have first hand experience with this subject after spending 2 long years working with a State level government agency to develop motor vehicle registration software ...

The problem is not so much about "how hard is it to write software that works" ... its more about "we're writing software for what we need RIGHT NOW".

When governments sit down to write software, its usually done through private contractors. So, a group of beaurocrats have a pow-wow and come up with a spec that generally reflects the type of work that the agency is doing "now", without much future consideration.

15 years later ... as legislation, beaurocracy, and agency regulations expand, so do the requirements of the software. For example, the Bureau of Motor Vehicles in an unspecified state put their first computer system in place in 1968. Since then, the scope of the BMV has expanded at least 10-fold.

Complicating the issue, "upgrades" are usually in the form of applying a new "layer" to the system somehow. As of 2003 in this unspecified state, the typical motor vehicle registration passes through 4 different systems before arriving in the central (OLD and limited) database at the state.

Complicating the problems even further are the many new layers of regulatory bloat -- meaning, the BMV is using software that met their needs in 1968, but doesn't meet their needs now. For example, (and this is how data goes bad), they're required to track whether or not somebody's registration is under suspension. However, back in 1968 registration suspension wasn't even a blip on the radar. To handle the problem after the "registration suspension legislation" was enacted, an "exception" had to be built into the system... if the street address field contains a special message, it indicates that the registration is under suspension. Ultimate problem... fields in the database are being used for purposes they were never intended. The age of the system does not allow for it to be updated properly.

Funny how you never hear ...

[Professor D]

The conspiracy theorists talk about how damned inefficient, bloated, clumsy and self-defeating government agencies projects are.

Somehow "they" have had UFO technology which would make petroleum obsolete since the '50s, conspired to kill JFK to keep it a secret, brainwashed Chapman to murder Lennon, created a secret government database tracking everyone's cash transactions, control us by putting chemicals in our water and thought patterns in satellite broadcasts. Oh yeah and "they" also were behind the 9/11 attacks as well.

Yet "they" can't even figure out how to keep track of whether or not foreign students went to class or not.

MS BS

[coyote-san]

That's MS BS. (And the cry of incompetent programmers for decades.) Even if we agree that all software has bugs - and I don't - that canard says nothing about all bugs being equal much less anything about all software having about the same number of bugs.

Any competent manager would know that experienced coders are usually FAR cheaper than inexperienced ones because they make fewer mistakes due to ignorance or indifference ("it works for me, so it's done!"). That gets you to the point of dealing with the more subtle and intrinsic bugs (e.g., due to conflicting requirements) quicker and cheaper, and the apparent cheaper cost of inexperienced developers is only achievable if you plan to release after coding is finished, not after testing is completed. Which is pretty much every MS *.0 release, now that I think about it -- got to get to market first, even if it's pure crap!

Oh yeah?

[egg troll]

The conspiracy theorists talk about how damned inefficient, bloated, clumsy and self-defeating government agencies projects are.

That's exactly what the CIA wants us to believe! Saaaaaay...aren't you the same Professor D who was involved with the faked moon landings!

EDS business model

[mysterious_mark]

Fixing broken EDS code is a large part of my job, the SEVIS project is no doubt another example of EDS shoddiness. The EDS business model seems to be as follows: - Collect $200/hr from client. - Pay h1-B $15/ hr to produce complete choss. - Management keeps the other $185/hr. for second vacation homes etc. But I suppose it is better that this project fail, at least we can count on EDS for something. MM

And another thing...

[Mattcelt]

I think there's an even darker side than what's being presented here in the brief - consider what happens when one of these 'glitches', whether techinical or PEBKAK, cause inaccurate information to be propogated through the linked government databases such as the TIA? Does joe traveller get strip-searched at every airport he goes to because someone "accidentally" put his name onto a terrorist watch list? Where does the government's responsibility to be accurate and precise with our information end? If the Credit Reporting Agencies are any indication, I think we have a potentially huge problem on our hands.

Re:And another thing...

[Aadain2001]

Add on top of that the tendency of this administration to make everything from the terrorist watch lists to what GW had in his morning coffee classified, and you won't even be able to ask for the information on yourself to make sure that it is accurate! And god help you if you do find an error and try to get it fixed!

No suprise...

[greppling]

...as the political decisions on this project misrespected one of most fundamental and easiest to understand rules of software project management: Never change the release date to be earlier than you originally intended.

After September 11, it was decided that the system had to be used starting early 2003. This was years earlier than in the original plans.

Re:Uh oh...

[jeffy124]

I have no experience whatsoever with SEVIS, but here a few tips I can think of on the fly:

-Xerox'd copies of any forms you've filled out related to all this. Carry these with you on the plane in carry-on luggage.

-Ask your advisor for a hard-copy listing of all the data that will be entered into the SEVIS system related to you.

-Consider having a letter written and signed by the advisor that entered your data into SEVIS, indicating what he/she has done. Also consider getting that advisor's office and home phone numbers in the event something goes wrong outside the 9-5 timeframe. Again, carry-on it.

-Call your advisor from Australia and ask that he/she check to ensure that your data is in the system before you leave.

There are probably a bunch of other things you can do, but the point I'm getting at is that you should try to cover all your bases and double check everything. Yeah, it's unfortunate you have to do this s---, but it might be the only way to prevent yourself from too much hassle.

System doesnt work

[t0ny]

I work with the government as a consultant, and I continue to be amazed at the waste and stupid use of money involved in government projects. What seems to be the rule, rather than the exception, is poorly skilled people pitching initiatives they have absolutely no skills to impliment, and they dont even put in the work to design it well.

There is one database used for payroll on which millions (if not tens of millions) were spent, and the end result is a system nobody is completely sure of, and which requires all the deparments involved to completely change all their procedures. And its even less flexible and problematic than what it is replacing. AND this is a custom application!

Until government starts paying tech people what they are worth in the private sector, you will always have poorly skilled bullshitters pedalling their wares to the public sector, who is suffering from the illusion that throwing money at a problem will make it go away.

Q:Elepehant? A:Mouse built to government spec.

[blhseawa]

This is the same old software engineering problem, over and over again.

A user who has never written a *COMPLETE* system specification, acutal has no idea what that is, who only knows what he/she does not want.

Software developers/coders/bodies who are not SME's (subject matter experts), making system / software decisions without either the knowledge or guidance to understand the ramifications of those decisions.

Neither users, nor software development companies want to deal with these issues, they would rather just get the money.

That is why most large software development/ service companies have such bad reputations.

According to SEI, (Software Engineering Insitute) over 70 per cent of all software development projects are terminated as failures.

Governments and Corporations - Clients or Children

[Onetus]

Disclaimer : I have worked for a number of Financial Institutions and Large Corporations.

My experience with the problems of these sorts of situations is as follows:
1. Sales droids underbid each other to get the job and commit to ludicrous time frames
2. Project teams end up with short development time and are always pushing to reach the deadlines in time.
3. Client changes their requirements, but will not change their expected delivery date. Either they refuse due to business need, or they do view their change as an actual change. More often they view their change as a "clarification" - even if it contradicts what their specifications orignially said.
4. Agressive job market has Project Managers kowtowing to Client demands.
5. Multiple departments are clients, but pay different amounts into pool. Each department seeks to maximise their benefit at the cost of other departments (despite fact part of same organisation - politics)

I mean, really, the problem exists in the fact business units will often not sit down and commit to producing clear, unambigious details of what they want & need. Bugs creep into the process when your dev's are working frantically to meet the deadlines and handle the unexpected change request.

And now a pithy little quote to put on your wall:
-----
Programming to Requirements is like walking on water.
It's easy to do when everything is frozen.

Re:Canada's National Gun Registry Anyone????

[El Cubano]

One billion (with a B) dollars

Canadain?

That would be like what, like US$ 150,000?

Bureaurcy and the lowest bidder

[bastion]

As a matter of public policy (and taxpayer money) is this level of non-functionality to be expected in these sorts of projects?

Yes.

Look on the bright side, if they can screw up simple projects like this how far do you really think TIA (Total Infomation Awareness) is gonna get?

TIA Alias Search: Commander Taco

Output: Mexican terrorist. Leader of collbration site for social dissidents.

Just far enough from the truth to get somebody sued....

It's the environment

[darnok]

I've worked in many government departments, always as an independent contractor. My experience is that government databases, and the apps that talk to these databases, are generally a mess.

I think there's a few issues involved:
- a lot of the code is written by people with skill levels that wouldn't be accepted in the outside world. I've got no doubt that there's highly skilful coders working for government, but there's a lot of duds as well and often your code is only as good as the weakest coder in the team. Furthermore, a lot of the weakest coders are the designated "experts" for legacy infrastructure, such as databases, and you find yourself having to rely on their input far more than you'd like
- a lot of government stuff has been outsourced, and generally the outsource partners seem to be less diligent for government contracts
- governments are forced to send everything out for tender, and may need to give work to the lowest bidder or face internal inquiries. In the private sector, you get to eliminate a lot of tender responses, either because you know the respondent is incompetent from past experience or the price is so low that the respondent couldn't possibly deliver a quality service; in government, you don't have the same luxury of filtering out the crap
- in the private sector, a lot of mainframers moved across to Unix, then Windows, in response to changing demand. While these people may not be expert in all 3 areas, they at least know enough to be able to hold a technical discussion with someone else from another area. In government, a lot of mainframers have been doing the same job forever, and you need to tie everything back to their narrow view of the world to get anything out of them whatsoever. You'd better not try to talk "Web Services" with these guys... There's a level of financial support and job security for these types of people in government that allows them to keep doing this, that simply doesn't exist in private enterprise

Now that I've pissed off everyone working in govt IT, I have to reiterate that some of them are extremely good. The issue is that the system doesn't work on a "survival of the fittest" system such as applies in private enterprise.

The key is in the rate structure

[plsuh]

I've worked on some government contracts, and in my opinion a big part of the problem is in the GSA schedule rate structure that the Federal uses for contractors. It is much more profitable for a contractor on a government project to put many junior people on a project rather than a few senior people, for the same amount of revenue. For instance, a junior developer may cost a contractor $50/hr with overhead, but the contractor is able to bill the government for that junior developer at $150/hr., a spread of $100/hr. A senior developer may cost $100/hr with overhead, but can only be billed to the government at $175/hr, a spread of only $75. Furthermore, the contractor can bill more hours of junior time than senior time under a given budget cap, compounding the effects of the greater spread. Thus, the incentives for contractors are to use as many junior developers as possible on a project, to increase the profit margin.

Unfortunately, It's a rule of thumb in this industry that a few good programmers are a lot more productive than many unskilled ones. The result is that many government IT projects are shoddily built by well-meaning but inexperienced developers who are put in that position by a contracting structure that fails to recognize the realities of the IT industry. Contractors are just responding rationally to the incentives that are presented to them.

These numbers are examples -- in fact the situation may be even worse. Federal government contracts vary in their rate structures, and many are stingier than this. It may well be impossible to bring on a senior developer as a subcontractor because the maximum hourly rate that the government will pay on a project is lower than the cost of the senior developer.

A prime contractor that I worked with staffed a large WebObjects project for the Department of Defense with a dozen or so low-paid, fresh out of community college drones. Every six months -- when a project review was due -- they would bring us on board as subcontractors for six to eight weeks. In that time, two or three of us would take the code base from where it was four months ago and bring it close enough to the required progress to get the contract renewed, and then the prime contractor would say "goodbye" and toss us out. Four months or so would pass by, with their people making little meaningful progress, and we would get a panicked call for six or eight weeks of more work to get by the next project review. (Did I mention that the prime contractor didn't pay the bills for one set of work until they needed us for the next project review? It got so bad that at one point we had to treat them as though their credit rating was zero, and demanded that payment for each week's worth of work be deposited in an escrow account before we would continue.)

By the way, this rate structure is not unique to government IT projects. Other types of government projects display the same professional services rate structure. When I worked for a (then) Big Six accounting firm as an economist, most consulting projects for corporate clients were staffed with a ratio of one partner and two or three senior managers to six or eight associates. However, the Federal government group was staffed with a ratio of one partner and one senior manager to twenty or so associates. I talked to the senior manager, and he told me that (a) the associates in the government contracting group were paid much less than we were on the corporate side since they billed out at a lower rate, and (b) the only way they could make money was to use lots of cheap associates because senior people could only break even at best at government rates.

Ya know, it'd be nice to see a GSA person squirm over this sort of thing in front of Congress some time. Then again, Congress may be part of the problem, as they'd rather generate lots of jobs for constituents, instead of a few.

--Paul

How hard is it to write software like this?

[dsplat]

I can think of several widely used systems that are backed by databases that work just fine. Slashdot wasn't built on a massive budget. Amazon doesn't have a history of "bleeding" data from one user to another. Google and Yahoo are certainly capable of handling tremendous loads.

I see several possible problems here. First, it is possible that this software was rushed into use before it was ready. Given the political pressures involved, I suspect that is part of the problem.

Second, I doubt that all of the programmers involved are of guru caliber. I don't intend to malign them. Even assuming that you have nothing but above-average programmers, when you build a huge project with lots of designers and coders, there are going to be miscommunications and some details that just aren't communicated.

Third, I would bet that this project has so much design documentation done up front that it is impossible for anyone to wrap their brains around the whole thing. This is, at best, a 1.0 release. And there are going to be design flaws in it. And the guys writing the code aren't likely to have a broad enough overview of it to spot them all. They also undoubtedly tripped over a lot of things that weren't specified up front and should have been. It is the nature of the game. But they weren't free to just choose a good solution when the questions came up.

The projects I cited at the beginning were developed by small teams with a vision of what they wanted to build. Within the constraints of the tools they had to work with and the general idea of what they were building, they were free to change the rules. They could refactor to their hearts' content. That is not going to be the case on huge government contracts.

Everything that we know about open source, agile/extreme programming, etc. doesn't apply to this kind of project.

All software has bugs

[Tony-A]

The thing is to be aware of this before coding and take steps to compensate and reduce the damage from those bugs.
An excuse after the fact, the canard seems infantile.

All software has bugs. They are most certainly not created equal. Just because there is no way to expose them, doesn't mean they're not there.

("it works for me, so it's done!")
OUCH!
There is a big difference between partially working for a few things for a few people and never failing for everything for everybody. Some of the critical knowledge lies in the application area and is neither achieved readily or even necessarily consistent.

programmers =~ lawyers

[Stephen Samuel]

'Of course you are right,' said the manager. 'But the key to achieving profitability and return on investment is to improve the development process, and with it the cost and quality of the end product. Software developers like to think they're doing something very special, but in fact it's an industrial process just like any other. The essence of software development is a quantitative approach to measuring and improving the performance of the software development process. What you don't measure you can't control.'

I think that, for most managers, the analogy which would most make sense to them is that Programmers are pretty much the same as Lawyers:

The goal, in either case, is to take a set of rules -- often arcane and ancient (programming languages and operating systems Vs. rules and laws), and combine them in such a way as to allow the client to achieve their wanted ends.
The judge would be the rough equivalent of a wetware execution unit.

Once they accept the lawyer analogy, then you can ask just how reasonable it would be to expect a lawyer to accomplish a lawsuit according to a tight schedule. Although it is doable, the tighter the schedule, the higher the price (often exponentially so).

I came up with this analogy because I ended up, a few years ago, self-representing myself in a reasonably complex lawsuit (It was about 4 years old by the time I got pulled in). With a couple of months heavy research I was able to do well enough in the courtroom (before the chief justice, and later at the Court of Appeals level) to reasonably impress just about every lawyer I dealt with in court.

I achieved this by pretty much applying my programming experience almost one-to-one. I simply treated the legalese and rules of court as a programming language. Old precedents were treated much like code snippits.

If you look at old slashdot postings, I think you can see that good programmers don't have that tough a time with laws and even court decisions. I submit that it's because the paradigms aren't really that different.

Brief Rebuttal

[Bios_Hakr]

I have been working in the USAF for about 8 years. 6 of that in WAN (longhaul voice and data), and 2 in Infrastructure and security. I'd like to offer another side to your story:

>but it needed to be done before the end of the fiscal year

This is how it works: The USAF has a budget. Each area gets a small slice. If filters down to each office having about $10k ~ $30k for operations that year. That money has to last all year. About 20% of that is kept in reserve funds. If that money is not needed by August, we are free to spend it. At that point, we develop a wish list and try to get that aproved. By time all this happens, we have about 5 weeks to spend the reserve money.

No one in the military likes it. All our contractors hate it. If you want it changed, write your congressperson and have them change 50+ years of bad management practices...

>The quality of code would've been greately improved if we coded, say 40 hrs/week instead of pulling all-nighters.

I have spent countless days and nights working overtime. So have a lot of my coworkers. In times of exercise or, God forbid, a war, we go to 12+ hour days. 6 days on and 1 day off are common during exercises.

Contractors always make fun of us for sloppy wiring, half-assed installs, unpatched servers, etc... When new equipment arrive, we usually have a few hours to determine where it will go and when. We are usually told that the old equipment stays in place until the new stuff is operational. This leads to massive misuse of rack space. and cluttered wiring.

Also, just like your code suffers from 40+ hours, my wiring suffers when I have to spend my Saturday morning connecting a new router.

No one likes to work overtime. Your work suffers just like mine. You may lose a contract because of your bad code. People could lose lives because of my bad wiring. Let's both work harder to keep our shit straight, regardless of hours worked.

>They insisted on .NET 2003 server with M$ SQL, etc., etc.

This is becuase we have a very nice license with MS for their stuff. We get good support, including semi-annual "Best Practices" reviews by MS inspectors. The US Gov paid for MS tools, we should use them. If you don't like it, write your congressperson. Personally, I'd love to be able to use Squid on Red Hat. Unfortunately, we don't have the money to spend on more software licenses after we bought MS stuff.

>asked if it would work with a win2003 server as opposed to a win2k

Our upgrade paths are fixed by MS. This absolutely sucks. Our systems require specific patch releases from MS. Once they stop supporting those patch paths, we have to upgrade. Agian, if you don't like it, write your congresscritter.

>but they didn't even know how to install windows

I'm throwing a bullshit flag on this play. I find it difficult to belive that no one knew how to install Windows. In the USAF, we have a NCC department that does nothing but install, configure, and maintain Win2k servers.

There may have been an internal power play based on getting Win2k3 server training. That is an ongoing military issue. Your boss tells you to do something. If you do it and screw it up, they ask if you were trained to do that thing. If you were not trained, then you go to federal-pound-me-in-the-ass prison for working on something without proper training. If you were trained and you screw it up, then you get in trouble for not folowing the training guidelines for whatever it was you broke.

Everyone working in a military NCC can install Win2k Workstation and Server. Many of them are MSCEs or higher. They could probably install Win2k3. They just wanted official training on that product before they tried something and broke it.

>Installing a a windows server is a mountain of a task for them.

No it isn't.

>Installing .NET is something that, as they say, they 'have been working on for

Re:Brief Rebuttal

[Osty]

This should have been in your install documentation anyway. Every contracted program I have ever dealt with has included install documentation. Most of it also includes referneces on how to install any required software. If your documentation needed revising, then admid to it. If they just needed to read the documantation, then tell them that. Don't just give someone a CD with a database and demand your paycheck.

Installation documentation is for the lazy. They should've had a proper MSI package that would install the database directly, without any intervention (well, maybe a gui on the installer asking a few easy-to-understand questions, or a few documented properties on the installer command line). Handing over a dump of a database, with or without documentation is weak, especially since it's hardly difficult to write an installer. Especially if you're smart about your database deployment (ie, consider detaching a clean copy of your development database, packaging up the .?df files, and re-attaching it to the production server via an MSI, rather than a shitload of files containing the schema that needs to be loaded).

It's 'Unit Tests', stupid!

[adamscottphotos]

I'm amazed that I haven't seen a single poster bring this up yet so...

The key to quality software; flexible, extensible, fault-tolerant, maintainable, and all of those other adjectives that 'good' software is supposed to have is very, very simple.
It's called Unit Testing. It's not brain surgery. I have worked on several medium to large scale projects (500k-3.5m lines) in several languages and environments, and I've yet to bomb one hard using this methodology, despite the usual client shenanigans.

Every time I write a functional subunit, I start by writing a series of tests (based on the spec, hopefully) that define 'doneness' for that subunit. Every object in the system has it's own set of tests. The test harnesses are chained together, so I can hit a button, so to speak, and run all of the hundreds to thousands of tests at once.

Whenever I check in new code changes, I run the test suite. If a test fails that previously worked, then I broke something. This plus good OOP practices (low coupling, high cohesion) allows you to make changes on the fly without the kind of 'The Money Pit' syndrome (fix one thing, another breaks) that is described in the article.

I am certain that the system in question was NOT developed with these methods. Most development organizations that I come into contact with pay lip service to the concept, but don't want to spent the perceived extra $ up front. The thought of all those developers writing TESTS when they could be writing CODE scares the willies out of them. But it pays for itself. It really does, every single time. Don't tell your boss, and try it on your next project. This is old news - google has a ton of info on it, and there are some good but unnecessary books also.

In the dot.com glory days, we had a huge system, running several hundred transasctions per second on a geographically distributed system of clients. We made fundamental architectural changes without a hitch, switched servers live without a hitch etc, and made a zillion little changes, all live, and all without a hitch (well, other than really stupid human errors, like locking out the client upgrade system with a bad password... oops). We had zero budget, and 2.5 developers. Unit tests are the Way, and if any company that doesn't mention them in your first meeting, run like hell.

Reactive, Just-In-Time Development

[prestidigital]

I've read article and many of the replies here. Several /.ers definitely describe flaws in government contracting processes and hiring practices that I've experienced, but I think they are missing the point. I think there is an additional, more fundamental flaw, that has been overlooked - or at least didn't get modded up high enough for me to see - maybe i should go trolling for a different set of opinions :^).

My experience tells me that the problems begin when we fall into the trap of trying to solve problems with a reactive mindset instead of a proactive mindset (proactive being favorable). We allow daunting problems and/or a need for revenue to back us into a corner time and time again and every time we are forced to hack our way out. Some of that is just old-fashioned survival, but a lot of it can be avoided with deliberate forethought, planning, discipline, and a commitment to quality and detail.

Avoiding clusterf*x has to be an institutional effort, whether the institution is a huge goverment agency or an tee-tiny, independent software shop. Everyone in the organization - operations, sales, IT - has to be on board with the policy that "if it's worth doing at all, then it's worth taking the time to do it right...the first time." I said "fundamental" earlier b/c that has to be something like lesson #5 in life's little handbook - we all heard it all too often when we were kids, we know it's true, and yet now we don't pay it any mind.

I do think the failure to heed that simple maxim usually starts in business development and snowballs by the time it gets to IT, but it really goes both ways. Everyone has to be responsible for maintaining the discipline required to produce quality.

What happened with this system is everyone involved got themselves all in a panic like a drowners who not only won't let you save them, but pull you under too. It's understandable given Sept 11, but "undertandable" and "right" are two different things. Legislators threw money at a situation they didn't try to understand. Deal-makers when after that money, promising to solve problems they knew they didn't understand. Developers enabled deal-makers by claiming to understand. No one took the time to do it right the first time.

P.S. It doesn't have to be this way.

It's not just one problem to solve

[Anonymous Coward]

Here's a quick bulleted list of items that I see as causing problems with the government projects I've been involved in.(yes, I've been doing reports for management waaaay toooo muuuch )

1. Requirements change over time, but inadequate funding available to really redo the design
2. The software was developed and designed for a 4 year life cycle and unfortunately we are in our 5th/10th/20th year of maintenance
3. Old software is compared to brand new commercial apps and their features, which didn't exist when the software was originally written. (Think of comparing a telnet, ASCII based application to a GUI app. And yes, ASCII based systems do still exist and are used for real work)
4. Use of cheap, inexerienced people in order to make a higher profit margin
5. Unable to charge a per usage/license/maintenance fee, only able to charge for the development costs up front, so potential profit avenues are reduced.
6. No integration planning on the government's side due to lack of expertise, politics or not realizing that different projects should be coordianted among contractors.
7. Eye Candy tools sold to higher ups without adequate technical input into the chosen packages.
8. "We've never done it that way before" uttered by company management, customers, developers, etc, which leads to long involved discussions and training basically as to why new Method A might be better than old Method B.

Now, the really, really hard part, what to do about it? I dunno, if I knew that, I'd be getting paid a lot more money then I am now and I have my underlings read /. to me while I sat by the pool. I am trying to do stuff, as I can, so that I can feel better about the projects I work on.

Myself, I am working on educating people in our company about other ways and means to make money from software and trying to insidiously ;-) get more competent people on the project in order to recover from past errors that are really biting us in an um, bad place. Progress is being made, but man, it is slow. (Think glaciers and you'll get an idea.)

You guys have only a vauge idea

[mess31173]

You guys have only a vague idea of how government works. Or if you do know, you have had vastly different experiences than I. I worked for the Dept of Revenue and Finance for a state government and this is exactly how things worked. Each government body, Natural Resources, Transportation, ect, ect each got a set budget based on how much they spent the year before and how much money the senate/governor allotted them from the total state budget that year. The big deal in our agency was to spend ALL of the money that was allotted to us by the end of the fiscal year otherwise next year we would lose what we didn't spend and our budget for next year would be shrank to the same ratio as what we spent the year before. Our budget would actually decrease if we didn't spend the all of the money in our budget!! Logic would tell me that if you can justify your budget but can manage it well and run UNDER your budget you would be REWARDED not the other way around. I digress. So, anyway, at the end of the year the director of the project would tell us to make a "wish list" (he actually called it that) and we got to buy pretty much anything we wanted, including contractors, and do anything we wanted. So, need a PDA? You got it, a new workstation? Got it. You name it, as long as we didn't deplete the pot we got it. Kind of crazy how the system encourages waste like that but it does.

I just live here....

DOI Websites Yanked *Again* for Security Flaws

[mattOzan]

Remember in late 2001 when the US Department of Interior was ordered by the court to take more than 100 of their web servers offline due to abysmal security? Hired white hats were easily able to gain access to the US Indian Trust database and found no security measures or even audit trails in place. Worried that this could be contributing to the agency's continuing mismanagement and loss of allegedly billions of dollars belonging to Native Americans, Judge Royce C. Lamberth ordered the DOI to "immediately shut down Internet access from any computer, server and system in the department that has access to individual Indian trust data."

The defense counsel noted that the fact that they took down over 100 mostly unrelated servers "...just shows you how inept they are. They don't even understand how these systems relate to each other so they just pull the plug on the entire system."

And now last month they were ordered to disconnect their servers again after refusing to let a court-appointed special master test the security measures they've supposedly put into place since then.

Sounds like an endemic problem for government agencies, at least at the federal level.

Ever try bidding on a government contract?

[putaro]

The reason the same gang of idiots keep getting work from the government is that they're the only people willing to bid on it. Government procurement is so bizarre that unless you have a team of specialists putting the bid proposal together you have no chance of getting it. Every large company I've worked at had a special "Federal Systems" division whose job it was to deal with that sillines.