Glitches in Massive Government Databases?

HBergeron asks: "Rather then post this as another YRO in the litany of new government datamarts there is a more fundamental question for all the coding Slashdot readers out there. This story, in Government Executive magazine, outlines the range of programming glitches in what is a relatively simple database. As a matter of public policy (and taxpayer money) is this level of non-functionality to be expected in these sorts of projects? Is the contractor just ripping off the taxpayers with bad code? How hard is it to write software like this that works?" The article focuses on the SEVIS database, but have others noticed similar trend in other government information systems?

Comments

All software has bugs

[ObviousGuy]

And the government system of going with the lowest bidder is bound to cause some problems as the more expensive engineers would no doubt bring better experience and know how with them. When you bring in the inexperienced because they are cheap, you frequently end up spending more in the long run than if you had paid for the expertise up front.

It's like they say, you get what you pay for. Cheap prices are only cheap if your time has no value.

Re:All software has bugs

[Xerithane]

It's like they say, you get what you pay for. Cheap prices are only cheap if your time has no value.

Wait! But what about Linux?

Time to end the sarcasm for the day..

Re:All software has bugs

[Tablizer]

And the government system of going with the lowest bidder is bound to cause some problems as the more expensive engineers would no doubt bring better experience and know how with them. When you bring in the inexperienced because they are cheap...

Inexperience may be part of it, but often government systems are subject to a lot of competing interest and tying together existing diverse systems such that simple requirements in isolation often balloon into complicated situations. As a contractor your hands are often tied WRT cleaning up existing bad processes and odd requirements to solve needs of competiting agencies, departments, etc.

It is often diplomatic issues that cause the messes, not technical ones.

Re:All software has bugs

[treat]

And the government system of going with the lowest bidder

If they go with the lowest bidder, why do they choose Microsoft over Redhat?

Re:All software has bugs

[aoteoroa]

To write perfect bug free software you must have a complete and accurate understanding of the end users problem. The best explanation I have found as to why this isn't as easy as it sounds was in a book called Software requirements and specifications which in the first chapter tells a story of a mathematician, finance director, manager, sociologist, and a stock broker discussing a recent failed project.

In 1993 the computer system project for the London Stock Exchange failed disastrously. 400 million pounds spent and nothing to show for it. Who was to pay? What had gone wrong? Why do so many developments end in disaster?

'Pure Ignorance,' said the mathematician. 'Software development is essentially a branch of mathematics. That is why computer science departments in universities have so often been closely associated with mathematics departments. You must understand that a program is a mathematical object. Its development is therefore a mathematical activity, of a particularly challenging kind. Those who engage in it should, of course, be competent both in using the appropriate mathematical notations and in drawing on the appropriate body of mathematical knowledge -that is, on knowledge of the relevant theorems. While we continue to ignore these facts we will continue to perpetrate disasters.'

'That's all very well,' said the finance director, 'but in my company we build systems to improve our business performance. I imagine that the Stock Exchange does the same. Software isn't mathematics: it's business. I think of a software development project as a capital investment. The test of its success is simply the value of the return on that investment to the company. The return in this case seems to be negative. The essential tools in a software project are financial risk analysis and discounted cash flow calculation'

'Of course you are right,' said the manager. 'But the key to achieving profitability and return on investment is to improve the development process, and with it the cost and quality of the end product. Software developers like to think they're doing something very special, but in fact it's an industrial process just like any other. The essence of software development is a quantitative approach to measuring and improving the performance of the software development process. What you don't measure you can't control.'

'But surely software development is done by people. And for people isn't it?' said the sociologist. 'Software is situated. You talk as if the system and its development were something objective, but really it has to be continually renegotiated subjectively between the various stakeholders, who all have their own agendas and perspectives. The success of any system depends directly on facilitating the negotiation, and on the determinant individual and group relationships in the societal context. I suspect that the Stock Exchange members belong to an authoritarian culture in which the dominant behaviour in inimical to peer group negotiation; perhaps that explains their failure.'

'This all seems ridiculous to me,' said the stockbroker. 'The plain fact is that the system was meant to serve the needs of brokers and jobbers of the Stock Exchange, and it didn't. It usually takes a professional working member of the exchange at least five years to learn how the Stock Exchange works, and I don't see why the analysts and programmers who make computer systems should expect to pick it up more quickly. A system for a particular business can only be built by people who are experts in that business. Domain knowledge, I think it is called. That's what matters.

Re:All software has bugs

[vladkrupin]

The quote from the original article:
How hard is it to write software like this that works?

Wow! Well said! My grandma couldn't have done better. In other words, please define 'works' for me. How many blue screens a day constitute 'works' and how many are too many?

Also, since we are at it, I want to reflect back on the latest project we have done. Incidentally, for the government. Before asking if a vendor is ripping the taxpayer off we need to consider how the government mismanages the resources it has. Consider the facts:

1. The project itself was fairly small and simple. I'd say it would normally take about 2 months to develop and deploy, but it needed to be done before the end of the fiscal year, so it was a 'now-or-never' situation, and was a horrible time-crunch. We had slightly more than half the time necessary to do it, but they won't even try to install it till probably the end of the year! The quality of code would've been greately improved if we coded, say 40 hrs/week instead of pulling all-nighters.

2. They tried to keep tabs on the development by scheduling 'technical meetings' over the phone. While there is nothing wrong with that per se, in a time-crunch that was a horrible waste of time. The smartest things we've heard from them were questions like 'Are you using hungarian notation?' or 'is your code well-documented?'....

3. They insisted on .NET 2003 server with M$ SQL, etc., etc. We did our best to make them consider PHP and the like, because that's what we normally use, but they were willing to pay extra to have that stuff developed in all-M$ stuff! We were told that the reason for that was because their IT was managing only M$ software, and the server was already there, and they couldn't have anything else (e.g. PHP). Fine, I can understand some bureaucracy in IT - that's cool, but imagine my surprise when, after we shipped them a CD with the project, they called us back and asked if it would work with a win2003 server as opposed to a win2k!!! Not only they didn't have the server yet, (or the infrastructure for that matter), but they didn't even know how to install windows! Which brings me to point #4:

4. Their IT is kick-ass. As in 'their ass needs to be kicked real hard'. Installing a a windows server is a mountain of a task for them. Installing .NET is something that, as they say, they 'have been working on for a while, but haven't got it quite yet'! And, when we give them a database dump they have no idea what to do with it and you have to walk them through the process (right-click on the 'Databases', select 'Create New database', click ok...) And they are paying these people!!! Errr... Let me re-phrase that - We are paying the government to employ those dumbheads! Thatnks goodness the network on which that is installed is not connected to the internet - the same idiots are in charge of security as well.

Yes, it is true that some contractors will rip off the government (and it is really the government's responsibility to make sure that doesn't happen! But that's not the point). The point is that even if they have a perfectly good product developed by honest people, they are still remarkably talented at screwing it up. Bureaucracy and lots of idiots in charge of hiring people are to blame.

Re:All software has bugs

[hendridm]

> And the government system of going with the lowest bidder is bound to cause some problems

I worked in a state agency, and the fact that we were required to take bids didn't really change who we purchased from. We just chose the vendor we liked best and justified it by writing the project needs around that vendor. They did that with employees too. When a new job opened, they often had someone in line for the position. However, equal opportunity required that the do interviews for the position. To justify the person they desired, they would write the job description and requirements based on the skills of the individual they liked. They would then schedule interviews even though they already had someone chosen for the position, just to meet requirements. I suppose they could have changed their mind if they found someone who was absolutely fabulous, but it's hard to convince an employer how great you are when in the back of their mind they don't think the interview is going to matter anyway.

Re:All software has bugs

[John Hasler]

> And the government system of going with the lowest
> bidder...

So you want them to give the contract to the most _highest_ bidder? The brother-in-law of the contract officer?

They give the contract to the lowest _qualified_ bidder. Doing otherwise would be stupid.

> It's like they say, you get what you pay for.

Bullshit.

Re:All software has bugs

[retto]

I've done some work with government organizations (not a lot so others way have had different experiences than mine) and from what I saw the biggest problem wasn't that the work was done by the lowest bidder, it was that the requirements were often created by people other than those that know the situation the best. Very little thought was given to designing something to be as usable and efficent as it can be, and more focus is given to making sure it gets finished by an arbitrary date. If it works, great, if not, it is ok as long as some department chief can say they are compliant with something by the required date. I've seen so many little problems that could have been fixed, but time involved in getting approval would have been more than then it was worth. I can't imagine the cumlative effect of all those little problems that get overlooked.

In the end you wind up with a big mess, tacking on or changing just enough to meet some kind of regulation. If you see something beyond the immediate scope of the project that would make things a lot easier and efficent, but it would require time/money or cooperation from someone else's department/division/agency it will be shot down as it won't be 'in the budget.'

Ok, that was my little rant. One time I had to sign a form to get the air conditioner turned on before the 'approved' time in a federal building, and I've been pissed off at bureaucracy ever since.

Re:All software has bugs

[Anonymous Coward]

not all windows shops and windows admins suck big giant donkey dicks.

but you DID just describe what has to be about 90% of the technical departments using MS.

Click "Next" on everything till you reach "Finish"

and know how to reboot the server.

that's about it...you are a qualified technician.

Re:All software has bugs

[Mr. Piddle]

Domain knowledge, I think it is called. That's what matters.

They are all right, except for the mathematician, of course (just kidding...well, mostly).

My own personal list of priorities would be (from high to non-existant):

1) Domain knowledge.
2) Solid data modeling ability.
3) Most everything else.
1,000,000) Buzzword compliance and/or magazine-cover conformance.

Unfortunately, most self-proclaimed prophets of the software world work in the exact opposite order, which is why nearly every project is a pain in the ass.

Re:All software has bugs

[big tex]

..As I step on my soapbox, donning the asbestos...

This, written by and modded "+5, Insightful" from the same group of programers that:

1) bitched about the NSPE coming down on CS's (emphasis on the science part of CS) calling themselves Engineers with a capital letter;
2) wonders why the IT market is softer than a sponge;
3) Actually debates the need for college education in CS;

and so on.

Now, I'm sure that there are quite a few excelent CS people who deserve the moniker "Engineer" (Linus comes to mind, occuping the other end of the spectrum from RMS).

As for the lowest bidder system, I work for a construction contractor. We build highways, bridges, and the like. We do the vast majority of our work under low bidder contracts. More importantly, we deliver the product on time and of a high quality.
How and why, you ask? Quite simply, because we must. We must because we said that we would. We must because the state deserves a fine highway for the money they give us. We must so that we can stay in business and make more money tomorrow. We must so that the system keeps working.

If you can't do what you do correctly, give it up. You are the people Ayn Rand warned us about.

Re:All software has bugs

[Unregistered]

While this was ment as a joke, i'd like to point out that people valuing time is the buisness model for all commercial linux vendors. You can get Red Hat free, but many companies pay lots of money for support.

Re:All software has bugs

[homebru]

We build highways, bridges, and the like. We do the vast majority of our work under low bidder contracts. More importantly, we deliver the product on time and of a high quality.

And how do you deal with the customer whose specs say (in effect) "just throw a log across that creek because all we need is a footpath for the weekend" and subsequently declare your work an extension to the InterState Highway System and in non-compliance (substandard) of rule Blah, section Blah-blah, part Blah-blah-blah, paragraph Blah-blah-blah-blah?

From the article: The pilot was not designed to become a national system, however. The INS had intended to examine its results and then build something new, school officials who participated in the test say. It was a "throwaway project," says Johnson. "It wasn't supposed to become something bigger."

This is one of the most common causes of failure that I have seen over the years. A refusal by management to see the difference between a "proof of concept" project and a "production" project.

Attention programmers. Learn this now and learn it well. There is no such thing as a "quick and dirty" project. Anything you write for hire can and probably will be pushed into production. And if you "assumed" that you could "get by" with single user code with (for example) no record locking, error testing, logging, transactioning, or provision for remote monitoring or backup, you just screwed the pooch. The minute you check that code into CVS, it's heading for production with hundreds of incompetent users who will expect 100.000% uptime. And management will quickly point you out as the author of the failing new product and your reputation is shot, your future with the company is shot, and you have given programmers everywhere another black eye. Gee, thanks.

People, what it is, is that every piece of code that you write for hire has to be the very best you can create. Because, while your customer may have only asked you to throw a one-log footbridge across the creek, s/he is expecting an eight-lane interstate highway structure.

Re:All software has bugs

[HiThere]

The problem is probably worse than just inexperience. I don't know what it is, and I work in the middle of it. Probably it's empire-building, or at least that's my guess.

Example: I have the database that I put together for my (small agency). It took a few months to do, but not many... perhaps three, and that's allowing for a bunch of mistakes. Then it was refined for a few years.

Three years ago we were told that a State Agency would be assuming the duty. They were legally obligated to begin about a year ago. They still aren't doing it. But I built it myself, but when I talk to the State people:
1) it's a team of people building it
2) the people don't stay at the same job
3) sometimes it's being done by people in a different department.

When I went to give them my data, they wouldn't even take an electronic file. They wanted typed forms. (I cheated and printed them out on a laser printer...but the spacing had to match that of a typewritten form.)

Now this was about 4,000 sheets of paper that they were going to need to enter by hand. My *guess* is that they were entering it into a 3270 based system (i.e., old IBM mainframe) and were then going to convert it into a PC based system. They were implementing it in FoxPro...probably. I don't really know. I never ended up talking to anyone who was doing the real work.

Personally, this was what I consider my "small test system". It's the one I traditionally converted from system to system during the process of learning the new system. I'm told that they'll be on-line with it in a month. (They'd better hurry, or I'll retire before they get it running... not that it matters.)

So I sure understand about the slowness, and I have a few theories as to what causes it: You need to take the cross-product of the Parkinson Laws with the "Mythical Man-Month". Then add a bunch of Dilbert for seasoning. Gourmet crusine to revolt the finest palate.

There are times when this is a good thing

[KPU]

If the government has a harder time keeping track of people, maybe it will be less ambitious. Never mind.

Surprising?

[bajo77]

This seems to be on par with other things the government tries to keep tabs on. They can't keep track of paroled felons, the database of people who can't vote is horribly flawed, and the soundex database that the airlines use doesn't work either.

Granted, this needs to change, but this isn't the first time the government has failed to provide adequate information regarding lists of people.

Re:Surprising?

[Guppy06]

"They can't keep track of paroled felons, the database of people who can't vote is horribly flawed,"

Then don't live in Florida. It was a system designed to try to help Florida local governments enforce a Florida state law. This has absolutely nothing to do with the federal government, which is what the federal constitution mandates.

The way you genericly complain about "the government" shows one of the problems in these information systems: a total lack of knowledge of how "the government" works here in the US. We don't have one monolithic government (yet), we have a federal government, 56 state/commonwealth/district/territorial governments, and 3,066 county/city/parish/borough governments. Each of them have different rights and responsibilities spelled out by constitutions and charters. Each of them operates in a slightly different manner from all the others to reflect the wants and needs of its people.

I'm reminded of a story here on Slashdot a few years back about how France's government went "online," giving internet users the ability to do things like get a driver's license or enroll in public schools on a website. There were a great many people who complained about how the US hasn't done anything like this, all by people who didn't seem to realize that just about everything this new French site did are the responsibilities of state and local governments here in the US.

The only way things like this will get easier is if the US shifts away from the federal republic model and become a monolithic republic ala France. This would involve disemboweling the federal constitution and burning all state constitutions. Personally, I'd rather not see that happen.

Taxpayer $?

[spumoni_fettuccini]

Is the contractor just ripping off the taxpayers with bad code?

They've been doing that for years. Toilet seats for $10,000; hammers for $7,000? Not only that how much money is wasted on the old "that's exactly what I asked for, but not what I wanted"?

Re:Taxpayer $?

[Brymouse]

You forget the 10,000 toliet seat was designed for use on bombers. You can use the toliet in sever turblance, and if the plane flips, you will not get shit all over yourself. In a combat situation, this is a bargian at 10k. Imagian if your taking a shit and you come under fire, with a convetional airline toliet you would be busy cleaning up, now that could cost you your life.

Re:Taxpayer $?

[Anonymous Coward]

That particular comment took a real lack of imagination. The reason the government pays $10,000 for toilet seats has less to do with being bad at finding low prices (The government is, suprisingly, relatively good at that, though perhaps less so at finding quality products for low prices.) than with the government being bad at, well, money laundering.

Say I have a project that's going to cost a little less than $100,000. It's something I can't get funding for because the Great Gods of Paperwork have not decreed it shall be so, possibly because it's a project which is a little shady. What's more, I'm high enough on the food chain to order my own toilet seats. (That is to say, I'm the head of my agency...) Thus, I order 10 toilet seats for $15/piece, but write them up as costing $10,000 each. Voila! My pet project is funded! As a bonus, the obnoxious news-coverage, should it materialize, is about run-of-the-mill government waste which everyone is used to and bored with, and so it goes away quickly, instead of being about whatever hare-brained unsavory scheme I actually used the money for, be it spying on Americans (the typical paranoia-novel stuff) or operating a government-owned strip-bar (oddly more likely...).

Let's face it. Government employees may include a lot of people who were B and C students in college, but you don't think that they actually pay $10K for toilet seats, do you?

Re:Taxpayer $?

[The-P]

I think something that is getting missed in this is that the government specs something, and when they spec it they don't mean buy it off the shelf (if they did they would just purchase it from a scheduled seller, not bid it out). Most of these $10,000 hammer, $7,000 toilet seat stories you hear are from NASA & aerospace bids. There is a reason these things cost ridiculous amounts of money. They are being made one at a time and they meet very stringent specifications.

P

Re:Taxpayer $?

[Anonymous Coward]

The toilet seat costs $9.99. The other $9,990.01 goes towards secret government programs like 'gell-coat' pretzels for easy swallowing, ultra fiber stain proof red dresses, and anti-nausea japanese food.

Re:Taxpayer $?

[spumoni_fettuccini]

The reason the government pays $10,000 for toilet seats has less to do with being bad at finding low prices

I spent time in Uncle Sam's Misguided Children, so yeah I knew there is/was fudging. Also [personal experience and no I was not in S4, logistics and supply] if a unit/department did not spend its entire allocated budget by the end of the fiscal year said budget was reduced. There were massive buying sprees of barracks supplies, spare parts, office furniture, etc. While not wasteful, unless it had a short shelf life, we wound up buying all kinds of crap that sat around for months before it was used. I'm not disagreeing with you; I'm just adding another expense theory.

Re:Taxpayer $?

[RALE007]

The government "spending" $10,000 per toilet seat, or $7,000 per hammer, is not due to some cunning shady "contractor" (vendor would be a better word) having duped poor innocent unsuspecting Uncle Sam into paying ridiculously inflated prices. The real reason(s) can be attributed to one of three likely candidates in my opinion:

1) A government employee(s) received a very large kickback from contractor (vendor) by accepting to pay the outrageously inflated prices.

2) The accounting department doesn't want their budget cut, so they happily pay inflated prices to "prove" they will need the same or greater funds next year.

(On a side note, the second reason happens all too often. I used to set up enterprise solutions for HP, eg tell IT departments what they need to buy from us to be able to do what they want. Half the time you talked to a government office, they had no project or system to set up, they just wanted to spend money to keep their budget.)

3) Government office accountant tacks on $6,995 to the actual $5 spent on a hammer to hide the funds he's been embezzling.

The situation is ridiculous, but you don't give the government enough credit for being a part of the problem. They're not only stupid, they're a bunch of dirty thieves too. Public servants my lily white....

Re:Taxpayer $?

[jratcliffe]

Not this again. Once again, the reason for the $500 hammers, etc., is a matter of cost accounting, not nefarious fraud. These "scandals" erupted because the cost accounting used for a number of government contracts spread the large overhead and R&D costs equally by product line item. In other words, if there were 1,000,000 items being purchased, and $500,000,000 in R&D costs, each item was assigned $500 in R&D cost, whether it was a $50,000 custom circuit board or a $10 hammer. A lousy way to do cost allocation? You betcha. Fraud? Nope. See this for more info.

Government Waste

[simsj]

This make me glad I don't pay taxes

Had to.

[Valar]

Neo: I just had a deja vu.
Morpheus: What? What did you see?
Neo: I saw the same Bush pass by twice.
Morpheus: Was it exactly the same Bush?
Neo: I dunno... could've been some kind of father son thing.
Morpheus: A deja vu is a glitch in the database. It usually happens when they change something. Particularly, votes.

decimal to blame?

[Thinkit3]

If everything were in hexadecimal and you didn't have to convert, I think that would solve this and many other problems.

EDS? Explains a lot...

[NickFitz]

EDS do a lot of systems that don't work, or don't work properly, or run massively over schedule and budget, here in the UK as well.

I just can't understand why governments insist on using them with the track record of cock-ups they have; they're not even cheap.

Re:EDS? Explains a lot...

[Onetus]

Actually they do a lot of systems that work. And work so well - they're never mentioned because there's no publicity when they just do their job. Since they're the founder of the IT Services industy and currently #2 in IT Services (after I.B.M.) they have done a lot of projects - worldwide.

If you want to believe everything the media reports and not read between the lines, go ahead. Just don't try and peddle it to the rest of us.

Re:EDS? Explains a lot...

[neitzsche]

I disagree with you. In my experience, every interface I have written to an EDS system has had to work around collosal conceptual errors on their part. YMMV.

Re:EDS? Explains a lot...

[zooblethorpe]

Okay, let's engage in a little thought exercise here. It's quite likely that EDS projects <insert epithet here> because they skimp on actual project implementation. Yet the money from the contracts must be going somewhere.

Hmm. Could it be they funnel some of the excess into lobbying and other less-tasteful forms of political influence?

In the blurred lines of the public-private sector,
((Product == Crap) && (Contract == Lucrative)) ? EngagePoliticalMachinery(Future = Lots more contracts) : CutCorners(Until(Contract == Lucrative))

What, me, cynical? Nah...
Apologies for cludgy code, but then hey, I'm a translator, not a developer. :P

25 Years of Government

[Grey_Coder]

I have been working for municipalities for 25 years. I have yet to see a major program work well or work at all without overruns. I have chalked it up to me lacking a MBA or Degree in Computer Science. I am just a poor hobbiest that thinks for a million or three you should get what you pay for. But like shrinked software there must be no implied warrantee or garentee it will work. Man I think for a couple million give me a couple coders and little hardware and sit back. Open source here we come.

This isn't really news

[dirtfirst]

...Except perhaps to the executives the magazine is aimed at. Early versions of software are generally pretty buggy, particularly if the target keeps changing, and most especially if it is in response to a hastily crafted law. The only thing that's surprising about this is that the output is taken so seriously by law enforcement officials *prior to completion*.

Don't they have some donuts that need eating?

'Lowest Tender' syndrome.

[The Famous Druid]

I've seen this sort of thing happen before.

Government departments are pretty-much obliged to go with the lowest tender, even if the people running the tender know that the winning bidders are a bunch of incompetents who couldn't organize a fsck in a brothel.

So, the lowest bid wins, and then even if they actually are well-meaning and try to do the right thing, they have such limited resourses that they usually have to resort to working too few staff too many hours.

The result will not be quality code.

What is this a surprise?

[mc6809e]

When it comes to government, failure is rewarded with more money. In fact, failure is often cited as proof not enough money is being spent.

Besides, why bother doing a good job if you know you'll get paid either way? That's what the tax collector is there for!

Anyone seen "Brazil"?

[the gnat]

I think this movie shows what a *real* totalitarian state would look like: the danger to our freedoms is not from corruption but from incompetence. Programs like TIA creep me out because I'm absolutely certain that the Feds will find a way to fuck it up and throw some poor guy in detention because the computer skipped a byte and came up with his name. Ditto for the PATRIOT Act. Few people have recognized this, but what's frightening about Ashcroft is not that he's a fundamentalist autocrat, but that he's an incompetent fool. If innocent people suffer from the government's extension of powers, it won't be due to the GOP taking out its enemies but because some FBI secretary got a virus on her computer.

I'm not a libertarian; the government indirectly pays a large portion of my salary. However, the extension of government power worries me, because the more control they have, the more opportunities to fuck our lives up.

Re:Anyone seen "Brazil"?

[qtp]

what's frightening about Ashcroft is not that he's a fundamentalist autocrat, but that he's an incompetent fool.

The real danger of John Ashcroft is that he's a fundamentalist autocrat and an incompetent fool.

Not suprising.

[Mr. Piddle]

How hard is it to write software like this that works?

Harder than you imagine. If you remove the pork-barrel politics, directives of what technology to use coming from the clouds, and the recently potty-trained project team members, there isn't much left to give the project a chance at success. Most of the project's time is probably spent learning the difference between JDBC and EJB or at meetings discussing the differences between JDBC and EJB. The remaining time is spent accomplishing little by discussing the well-presented but vacuous system requirements for the project. Whenever I see a job posting for a database project for a government agency, I pass it and look for projects worth doing. If it mentions .NET or J2EE, I pass it by doubly fast.

I don't like this conclusion, but I've worked on, interviewed for, or heard about enough of these projects to realize they are pretty much all the same and for what seems to be all the wrong reasons.

its about "now"

[cr@ckwhore]

I have first hand experience with this subject after spending 2 long years working with a State level government agency to develop motor vehicle registration software ...

The problem is not so much about "how hard is it to write software that works" ... its more about "we're writing software for what we need RIGHT NOW".

When governments sit down to write software, its usually done through private contractors. So, a group of beaurocrats have a pow-wow and come up with a spec that generally reflects the type of work that the agency is doing "now", without much future consideration.

15 years later ... as legislation, beaurocracy, and agency regulations expand, so do the requirements of the software. For example, the Bureau of Motor Vehicles in an unspecified state put their first computer system in place in 1968. Since then, the scope of the BMV has expanded at least 10-fold.

Complicating the issue, "upgrades" are usually in the form of applying a new "layer" to the system somehow. As of 2003 in this unspecified state, the typical motor vehicle registration passes through 4 different systems before arriving in the central (OLD and limited) database at the state.

Complicating the problems even further are the many new layers of regulatory bloat -- meaning, the BMV is using software that met their needs in 1968, but doesn't meet their needs now. For example, (and this is how data goes bad), they're required to track whether or not somebody's registration is under suspension. However, back in 1968 registration suspension wasn't even a blip on the radar. To handle the problem after the "registration suspension legislation" was enacted, an "exception" had to be built into the system... if the street address field contains a special message, it indicates that the registration is under suspension. Ultimate problem... fields in the database are being used for purposes they were never intended. The age of the system does not allow for it to be updated properly.

Re:its about "now"

[jafac]

It's even worse when COTS software and hardware are involved.

If the project takes 5 years to complete (propose, fund, define, develop, test, implement), you end up with a system that's 5 years out of date. You'll have a bunch of desktop systems with Pentium 90 CPU's, running Windows 95 at best.

And getting any improvements implemented literally takes an act of congress (or "creative accounting" - which several of our larger contractors have been bitchslapped for in recent years).

It's okay in the Military space, because we're still 30 years ahead of any potential competitors technology-wise.
But with computers - the backwardsness is blindingly obvious to someone who has an up-to-date system in their own home.

Maybe the problem is - we shouldn't be paying government employees enough money that they can buy better equipment for themselves at home, than we can afford to supply them with for their jobs?

Funny how you never hear ...

[Professor D]

The conspiracy theorists talk about how damned inefficient, bloated, clumsy and self-defeating government agencies projects are.

Somehow "they" have had UFO technology which would make petroleum obsolete since the '50s, conspired to kill JFK to keep it a secret, brainwashed Chapman to murder Lennon, created a secret government database tracking everyone's cash transactions, control us by putting chemicals in our water and thought patterns in satellite broadcasts. Oh yeah and "they" also were behind the 9/11 attacks as well.

Yet "they" can't even figure out how to keep track of whether or not foreign students went to class or not.

Canada's National Gun Registry Anyone????

[Myrv]

Look no further than the fiasco with Canada's National Gun Registry to see how badly the government can screw up a database project.

After living through that I no longer have any confidence at all in the government to be able to implement any IT project competently. One billion (with a B) dollars to develop a database that essentially matches a gun serial number to a name and address. And they're still not done. HOW? Somebody please explain that one to me...

Re:Canada's National Gun Registry Anyone????

[El Cubano]

One billion (with a B) dollars

Canadain?

That would be like what, like US$ 150,000?

Re:Canada's National Gun Registry Anyone????

[SoSueMe]

Heh, I was going to say "What's that? a buck twenty?".

Actually, it's more like 733,360,738.55 USD.

You Yanks pay that for Pentagon toilet seats, right?

Re:Canada's National Gun Registry Anyone????

[demaria]

At least when we overpay for shit, we shit on it. :-D

MS BS

[coyote-san]

That's MS BS. (And the cry of incompetent programmers for decades.) Even if we agree that all software has bugs - and I don't - that canard says nothing about all bugs being equal much less anything about all software having about the same number of bugs.

Any competent manager would know that experienced coders are usually FAR cheaper than inexperienced ones because they make fewer mistakes due to ignorance or indifference ("it works for me, so it's done!"). That gets you to the point of dealing with the more subtle and intrinsic bugs (e.g., due to conflicting requirements) quicker and cheaper, and the apparent cheaper cost of inexperienced developers is only achievable if you plan to release after coding is finished, not after testing is completed. Which is pretty much every MS *.0 release, now that I think about it -- got to get to market first, even if it's pure crap!

Not necessarily bugs

[BigBadBri]

Firstly, it is offensive that the targets of surveillance have to pay for it.

That said, the 'bug' mentioned (that of getting a non-related set of students) smacks of shit data entry rather than a programming failure.

Lord help you, America, if you acquiesce to this sort of intrusive crap.

Before you know it, you'll be as bad as Europe, and soon after, as bad as Britain.

You'll be being persecuted on the basis of bad data, bad laws and stupid politicians.

That's our job.

Stop it.

Oh yeah?

[egg troll]

The conspiracy theorists talk about how damned inefficient, bloated, clumsy and self-defeating government agencies projects are.

That's exactly what the CIA wants us to believe! Saaaaaay...aren't you the same Professor D who was involved with the faked moon landings!

Sounds like poor relational design

[GFW]

Unfortunately, no technical details were given (such as programming language(s), database system, and general architecture type. However, a number of the bugs (like the one about not being able to graduate from a BA to Masters, and the one about the birth of a child) suggest an underlying poor relational design. If I knew more about the overall architecture, I could comment more about how many bugs one would expect. Certainly, other people have commented that you can have lots of bugs *before* you release and start taking the program seriously... I understand EDS has often gotten away with shoddy work for the government. I don't know why (political payoffs).

Foreign students KNOW

[Anonymous Coward]

that "sévices" in French is pronounced "sevis", and means "cruelty".

EDS business model

[mysterious_mark]

Fixing broken EDS code is a large part of my job, the SEVIS project is no doubt another example of EDS shoddiness. The EDS business model seems to be as follows: - Collect $200/hr from client. - Pay h1-B $15/ hr to produce complete choss. - Management keeps the other $185/hr. for second vacation homes etc. But I suppose it is better that this project fail, at least we can count on EDS for something. MM

This is good

[frovingslosh]

Is the contractor just ripping off the taxpayers with bad code? How hard is it to write software like this that works?"

Of course it's a taxpayer ripoff. Thank goodness we're not getting all the government we're paying for.

And another thing...

[Mattcelt]

I think there's an even darker side than what's being presented here in the brief - consider what happens when one of these 'glitches', whether techinical or PEBKAK, cause inaccurate information to be propogated through the linked government databases such as the TIA? Does joe traveller get strip-searched at every airport he goes to because someone "accidentally" put his name onto a terrorist watch list? Where does the government's responsibility to be accurate and precise with our information end? If the Credit Reporting Agencies are any indication, I think we have a potentially huge problem on our hands.

Re:And another thing...

[Aadain2001]

Add on top of that the tendency of this administration to make everything from the terrorist watch lists to what GW had in his morning coffee classified, and you won't even be able to ask for the information on yourself to make sure that it is accurate! And god help you if you do find an error and try to get it fixed!

No suprise...

[greppling]

...as the political decisions on this project misrespected one of most fundamental and easiest to understand rules of software project management: Never change the release date to be earlier than you originally intended.

After September 11, it was decided that the system had to be used starting early 2003. This was years earlier than in the original plans.

My guess...

[adagioforstrings]

If it's anything like my work, it probably started out as an Access (etc) database created by someone with limited knowledge of database design to try to make their job easier. It worked well enough and someone decided to get a contractor to flesh it out (perhaps explaining why the contractor referred questions back to the govnt: they are limited by using existing database or something). I don't think the contractor is necessarily ripping off the taxpayers, but who knows. Still, you'd think they'd do a better job of it. I think it's something started small, but got too big too fast. Happens all the time in companies, but has bigger impact when it happens in the government.

Re:Uh oh...

[jeffy124]

I have no experience whatsoever with SEVIS, but here a few tips I can think of on the fly:

-Xerox'd copies of any forms you've filled out related to all this. Carry these with you on the plane in carry-on luggage.

-Ask your advisor for a hard-copy listing of all the data that will be entered into the SEVIS system related to you.

-Consider having a letter written and signed by the advisor that entered your data into SEVIS, indicating what he/she has done. Also consider getting that advisor's office and home phone numbers in the event something goes wrong outside the 9-5 timeframe. Again, carry-on it.

-Call your advisor from Australia and ask that he/she check to ensure that your data is in the system before you leave.

There are probably a bunch of other things you can do, but the point I'm getting at is that you should try to cover all your bases and double check everything. Yeah, it's unfortunate you have to do this s---, but it might be the only way to prevent yourself from too much hassle.

System doesnt work

[t0ny]

I work with the government as a consultant, and I continue to be amazed at the waste and stupid use of money involved in government projects. What seems to be the rule, rather than the exception, is poorly skilled people pitching initiatives they have absolutely no skills to impliment, and they dont even put in the work to design it well.

There is one database used for payroll on which millions (if not tens of millions) were spent, and the end result is a system nobody is completely sure of, and which requires all the deparments involved to completely change all their procedures. And its even less flexible and problematic than what it is replacing. AND this is a custom application!

Until government starts paying tech people what they are worth in the private sector, you will always have poorly skilled bullshitters pedalling their wares to the public sector, who is suffering from the illusion that throwing money at a problem will make it go away.

Q:Elepehant? A:Mouse built to government spec.

[blhseawa]

This is the same old software engineering problem, over and over again.

A user who has never written a *COMPLETE* system specification, acutal has no idea what that is, who only knows what he/she does not want.

Software developers/coders/bodies who are not SME's (subject matter experts), making system / software decisions without either the knowledge or guidance to understand the ramifications of those decisions.

Neither users, nor software development companies want to deal with these issues, they would rather just get the money.

That is why most large software development/ service companies have such bad reputations.

According to SEI, (Software Engineering Insitute) over 70 per cent of all software development projects are terminated as failures.

This is terrorism!

[IdleTime]

This is ofcourse the result of terrorists!
Much easier to wreck havoc on a government project and cause disruption through buggy software than to take the time to learn to fly and then hijacka nd hit public buildings. The most positive thing is ofcourse that you, the terrorist, is not killed :)

If the US really wants to get rid of all potential terrorists, they have to evict everyone of a certain religious and cultural group and then close the borders and let no-one in and out. This is ofcourse not possible, hence all these measures taken by the gov is virtually useless.

Governments and Corporations - Clients or Children

[Onetus]

Disclaimer : I have worked for a number of Financial Institutions and Large Corporations.

My experience with the problems of these sorts of situations is as follows:
1. Sales droids underbid each other to get the job and commit to ludicrous time frames
2. Project teams end up with short development time and are always pushing to reach the deadlines in time.
3. Client changes their requirements, but will not change their expected delivery date. Either they refuse due to business need, or they do view their change as an actual change. More often they view their change as a "clarification" - even if it contradicts what their specifications orignially said.
4. Agressive job market has Project Managers kowtowing to Client demands.
5. Multiple departments are clients, but pay different amounts into pool. Each department seeks to maximise their benefit at the cost of other departments (despite fact part of same organisation - politics)

I mean, really, the problem exists in the fact business units will often not sit down and commit to producing clear, unambigious details of what they want & need. Bugs creep into the process when your dev's are working frantically to meet the deadlines and handle the unexpected change request.

And now a pithy little quote to put on your wall:
-----
Programming to Requirements is like walking on water.
It's easy to do when everything is frozen.

Legacy systems

[shogarth]

I can see how the individual campuses could be in the hurt locker. For obvious reasons, they are not going to want to do manual updates every time an address changes. That means integration with existing systems that (hopefully) allow students/faculty/staff to update their own records.

Many campuses (at least state campuses) are running really old student records systems. Imagine that the web front end for the users is built from screen-scrapes of old mainframe applications. To make it worse, now try to interface 20+-year-old, non-relational, pre-SQL student records systems with a new distributed database model.

The result is that the schools will have to reallocate senior programmer (who else will know how the records system works) time bumping other, mission-oriented tasks in the name of National Security.

Wonder what the SEVIS site is running?

[SysKoll]

A sentence in the article attracted my attention:

Sometimes, SEVIS crashed under the stress and expunged the day's work. The delays and headaches led some schools to close their student offices and ask employees to work nights and weekends, when traffic was lighter.

That's as good as putting a logo on it. So I went to Netcraft and checked the SAVIS web site, egov.immigration.gov. And sure enough: The site egov.immigration.gov is running Microsoft-IIS/5.0 on Windows 2000

I bet that the database is also running on Win2K...

Oh golly, you mean that when you put a high-volume site under Windows, you get (gasp) crashes and data losses? No way! Who would have thought that? Obviously, the poor Dept of Justice was a victim of an unheard-of, unexpected problem.

Not.

-- SysKoll

Re:Wonder what the SEVIS site is running?

[pinko-rat-bastard]

I was wondering that myself, especially after reading this quote from the article:

The association notes an anonymous report of technicians telling one school that a "catastrophic failure" had occurred while processing its records. When school officials asked what had caused the error, and what "catastrophic" meant, the technicians said they didn't know.

It may just be a coincidence, but this is (literally) an error message sometimes reported by ADO (Microsoft's Active Data Objects)

Re:Nobody read this

[captain_craptacular]

Actually if YOU read the article you would know that that quote refers to an earlier prototype that piloted in the mid-late 80's and was scrapped. The current SEVIS system is not and was never a prototype, it is and always has been a full production system, albiet a crappy one.

Bureaurcy and the lowest bidder

[bastion]

As a matter of public policy (and taxpayer money) is this level of non-functionality to be expected in these sorts of projects?

Yes.

Look on the bright side, if they can screw up simple projects like this how far do you really think TIA (Total Infomation Awareness) is gonna get?

TIA Alias Search: Commander Taco

Output: Mexican terrorist. Leader of collbration site for social dissidents.

Just far enough from the truth to get somebody sued....

Old news.

[afidel]

The nuclear materials database was found to suffer from flaws in Microsoft's SQL server back in 2001. This series of flaws led to the amount of tracked materials being incorrectly reported although the actual information was being tracked correctly. The origional slashdot article can be found Here.

It's the environment

[darnok]

I've worked in many government departments, always as an independent contractor. My experience is that government databases, and the apps that talk to these databases, are generally a mess.

I think there's a few issues involved:
- a lot of the code is written by people with skill levels that wouldn't be accepted in the outside world. I've got no doubt that there's highly skilful coders working for government, but there's a lot of duds as well and often your code is only as good as the weakest coder in the team. Furthermore, a lot of the weakest coders are the designated "experts" for legacy infrastructure, such as databases, and you find yourself having to rely on their input far more than you'd like
- a lot of government stuff has been outsourced, and generally the outsource partners seem to be less diligent for government contracts
- governments are forced to send everything out for tender, and may need to give work to the lowest bidder or face internal inquiries. In the private sector, you get to eliminate a lot of tender responses, either because you know the respondent is incompetent from past experience or the price is so low that the respondent couldn't possibly deliver a quality service; in government, you don't have the same luxury of filtering out the crap
- in the private sector, a lot of mainframers moved across to Unix, then Windows, in response to changing demand. While these people may not be expert in all 3 areas, they at least know enough to be able to hold a technical discussion with someone else from another area. In government, a lot of mainframers have been doing the same job forever, and you need to tie everything back to their narrow view of the world to get anything out of them whatsoever. You'd better not try to talk "Web Services" with these guys... There's a level of financial support and job security for these types of people in government that allows them to keep doing this, that simply doesn't exist in private enterprise

Now that I've pissed off everyone working in govt IT, I have to reiterate that some of them are extremely good. The issue is that the system doesn't work on a "survival of the fittest" system such as applies in private enterprise.

and by the way....

[qtp]

Innocent people do suffer when the GOP is "taking out its enemies".

The key is in the rate structure

[plsuh]

I've worked on some government contracts, and in my opinion a big part of the problem is in the GSA schedule rate structure that the Federal uses for contractors. It is much more profitable for a contractor on a government project to put many junior people on a project rather than a few senior people, for the same amount of revenue. For instance, a junior developer may cost a contractor $50/hr with overhead, but the contractor is able to bill the government for that junior developer at $150/hr., a spread of $100/hr. A senior developer may cost $100/hr with overhead, but can only be billed to the government at $175/hr, a spread of only $75. Furthermore, the contractor can bill more hours of junior time than senior time under a given budget cap, compounding the effects of the greater spread. Thus, the incentives for contractors are to use as many junior developers as possible on a project, to increase the profit margin.

Unfortunately, It's a rule of thumb in this industry that a few good programmers are a lot more productive than many unskilled ones. The result is that many government IT projects are shoddily built by well-meaning but inexperienced developers who are put in that position by a contracting structure that fails to recognize the realities of the IT industry. Contractors are just responding rationally to the incentives that are presented to them.

These numbers are examples -- in fact the situation may be even worse. Federal government contracts vary in their rate structures, and many are stingier than this. It may well be impossible to bring on a senior developer as a subcontractor because the maximum hourly rate that the government will pay on a project is lower than the cost of the senior developer.

A prime contractor that I worked with staffed a large WebObjects project for the Department of Defense with a dozen or so low-paid, fresh out of community college drones. Every six months -- when a project review was due -- they would bring us on board as subcontractors for six to eight weeks. In that time, two or three of us would take the code base from where it was four months ago and bring it close enough to the required progress to get the contract renewed, and then the prime contractor would say "goodbye" and toss us out. Four months or so would pass by, with their people making little meaningful progress, and we would get a panicked call for six or eight weeks of more work to get by the next project review. (Did I mention that the prime contractor didn't pay the bills for one set of work until they needed us for the next project review? It got so bad that at one point we had to treat them as though their credit rating was zero, and demanded that payment for each week's worth of work be deposited in an escrow account before we would continue.)

By the way, this rate structure is not unique to government IT projects. Other types of government projects display the same professional services rate structure. When I worked for a (then) Big Six accounting firm as an economist, most consulting projects for corporate clients were staffed with a ratio of one partner and two or three senior managers to six or eight associates. However, the Federal government group was staffed with a ratio of one partner and one senior manager to twenty or so associates. I talked to the senior manager, and he told me that (a) the associates in the government contracting group were paid much less than we were on the corporate side since they billed out at a lower rate, and (b) the only way they could make money was to use lots of cheap associates because senior people could only break even at best at government rates.

Ya know, it'd be nice to see a GSA person squirm over this sort of thing in front of Congress some time. Then again, Congress may be part of the problem, as they'd rather generate lots of jobs for constituents, instead of a few.

--Paul

Not confined to the public sector

[Polymath Crowbane]

Sadly, I've seen plenty of project disasters like this in the private sector. It should be a no-brainer that such a project starts with an understanding of the existing process and a vision of the new process. This leads to a written description of the process, including the collection, manipulation and display of data. From here, one can create a data model that describes and supports those needs.

All too often, what happens is that someone either sits down and knocks out a database out of thin air (with no understanding of the underlying business needs), or a team sits down with the latest and greatest methodology and tool set and forces the process into their pet methodology and tool set (again, with no understanding of the underlying business needs). I don't think perceptive /.ers need to be told what happens next.

But why does this happen? IMHO, we have dismissed the value of generalists, those who understand the underlying business model, the people and processes involved and the technical means available. At best, most teams have a business expert, a development expert, a database expert and a project manager, who likely does not have business expertise and, hence, cannot act as a proper liaison between the business and technical sides.

The generalist, being neither fish now fowl, and being more mature/older to boot, tends to be the target of early retirement and layoffs. And yet, these are the very people most needed by both public and private sectors. They are the ones who can make sense of the customer spec and present it to the team in a fashion that makes sense. They are the ones who can explain development realities to the business side in language that they can understand.

Does this mske sense, or is it the nostalgic longing of a generalist?

How hard is it to write software like this?

[dsplat]

I can think of several widely used systems that are backed by databases that work just fine. Slashdot wasn't built on a massive budget. Amazon doesn't have a history of "bleeding" data from one user to another. Google and Yahoo are certainly capable of handling tremendous loads.

I see several possible problems here. First, it is possible that this software was rushed into use before it was ready. Given the political pressures involved, I suspect that is part of the problem.

Second, I doubt that all of the programmers involved are of guru caliber. I don't intend to malign them. Even assuming that you have nothing but above-average programmers, when you build a huge project with lots of designers and coders, there are going to be miscommunications and some details that just aren't communicated.

Third, I would bet that this project has so much design documentation done up front that it is impossible for anyone to wrap their brains around the whole thing. This is, at best, a 1.0 release. And there are going to be design flaws in it. And the guys writing the code aren't likely to have a broad enough overview of it to spot them all. They also undoubtedly tripped over a lot of things that weren't specified up front and should have been. It is the nature of the game. But they weren't free to just choose a good solution when the questions came up.

The projects I cited at the beginning were developed by small teams with a vision of what they wanted to build. Within the constraints of the tools they had to work with and the general idea of what they were building, they were free to change the rules. They could refactor to their hearts' content. That is not going to be the case on huge government contracts.

Everything that we know about open source, agile/extreme programming, etc. doesn't apply to this kind of project.

All software has bugs

[Tony-A]

The thing is to be aware of this before coding and take steps to compensate and reduce the damage from those bugs.
An excuse after the fact, the canard seems infantile.

All software has bugs. They are most certainly not created equal. Just because there is no way to expose them, doesn't mean they're not there.

("it works for me, so it's done!")
OUCH!
There is a big difference between partially working for a few things for a few people and never failing for everything for everybody. Some of the critical knowledge lies in the application area and is neither achieved readily or even necessarily consistent.

programmers =~ lawyers

[Stephen Samuel]

'Of course you are right,' said the manager. 'But the key to achieving profitability and return on investment is to improve the development process, and with it the cost and quality of the end product. Software developers like to think they're doing something very special, but in fact it's an industrial process just like any other. The essence of software development is a quantitative approach to measuring and improving the performance of the software development process. What you don't measure you can't control.'

I think that, for most managers, the analogy which would most make sense to them is that Programmers are pretty much the same as Lawyers:

The goal, in either case, is to take a set of rules -- often arcane and ancient (programming languages and operating systems Vs. rules and laws), and combine them in such a way as to allow the client to achieve their wanted ends.
The judge would be the rough equivalent of a wetware execution unit.

Once they accept the lawyer analogy, then you can ask just how reasonable it would be to expect a lawyer to accomplish a lawsuit according to a tight schedule. Although it is doable, the tighter the schedule, the higher the price (often exponentially so).

I came up with this analogy because I ended up, a few years ago, self-representing myself in a reasonably complex lawsuit (It was about 4 years old by the time I got pulled in). With a couple of months heavy research I was able to do well enough in the courtroom (before the chief justice, and later at the Court of Appeals level) to reasonably impress just about every lawyer I dealt with in court.

I achieved this by pretty much applying my programming experience almost one-to-one. I simply treated the legalese and rules of court as a programming language. Old precedents were treated much like code snippits.

If you look at old slashdot postings, I think you can see that good programmers don't have that tough a time with laws and even court decisions. I submit that it's because the paradigms aren't really that different.

Re:programmers =~ lawyers

[ctve]

If I had some points, I'd mod it.

It's quite interesting to me, as I've been thinking about what attributes I got from my parents. My father is a lawyer, and I became a programmer, and I also have an amateur interest in the law.

Both are based on rules, and both are based upon a certain degree of exactitude. One thing that I notice that many people I know don't is when a law is badly drafted and ambiguous, as I think it comes from the part of my brain that sees badly written and ambiguous specifications.

Pleasantly Surprised

[paxil]

I must say that while working at a VA hospital last year I was very much impressed with their computer system.

The VA is the largest healthcare organization in the US, so they have the resources to build their own system. Contrary to what I was expecting, it is intuitive, just plain works, and IMHO blows away the stuff from Cerner or Meditech.

They have been working on it for twenty years, so it has the advantage of maturity, but even the newer bits such as windows interfaces running on Citrix are nice and stable.

Some background on the system can be found here.

Seems that is is mostly implemented in MUMPS so they score poorly in the buzzword complience department.

Anyways, I was just surprised that the government sometimes does seem to "get it".

What about Linux???

[Nick Driver]

Wait! But what about Linux?

Time to end the sarcasm for the day..


Linux is not necessarily "cheap". It requires a substantial investment of time and effort in learning how to install, configure and use it well. My time is worth a lot of money. And I have very much gotten great value for my investment of time in learning Linux. In the end, Linux is certainly a bargain, but it ain't "cheap".

Re:What about Linux???

[Lord Ender]

And if Linux were easy to install and configure, it would take less of your time, and therefore be cheaper. By that reasoning, the better Linux the cheaper, and hence worse (get what you pay for) it is. Hrmm.

Brief Rebuttal

[Bios_Hakr]

I have been working in the USAF for about 8 years. 6 of that in WAN (longhaul voice and data), and 2 in Infrastructure and security. I'd like to offer another side to your story:

>but it needed to be done before the end of the fiscal year

This is how it works: The USAF has a budget. Each area gets a small slice. If filters down to each office having about $10k ~ $30k for operations that year. That money has to last all year. About 20% of that is kept in reserve funds. If that money is not needed by August, we are free to spend it. At that point, we develop a wish list and try to get that aproved. By time all this happens, we have about 5 weeks to spend the reserve money.

No one in the military likes it. All our contractors hate it. If you want it changed, write your congressperson and have them change 50+ years of bad management practices...

>The quality of code would've been greately improved if we coded, say 40 hrs/week instead of pulling all-nighters.

I have spent countless days and nights working overtime. So have a lot of my coworkers. In times of exercise or, God forbid, a war, we go to 12+ hour days. 6 days on and 1 day off are common during exercises.

Contractors always make fun of us for sloppy wiring, half-assed installs, unpatched servers, etc... When new equipment arrive, we usually have a few hours to determine where it will go and when. We are usually told that the old equipment stays in place until the new stuff is operational. This leads to massive misuse of rack space. and cluttered wiring.

Also, just like your code suffers from 40+ hours, my wiring suffers when I have to spend my Saturday morning connecting a new router.

No one likes to work overtime. Your work suffers just like mine. You may lose a contract because of your bad code. People could lose lives because of my bad wiring. Let's both work harder to keep our shit straight, regardless of hours worked.

>They insisted on .NET 2003 server with M$ SQL, etc., etc.

This is becuase we have a very nice license with MS for their stuff. We get good support, including semi-annual "Best Practices" reviews by MS inspectors. The US Gov paid for MS tools, we should use them. If you don't like it, write your congressperson. Personally, I'd love to be able to use Squid on Red Hat. Unfortunately, we don't have the money to spend on more software licenses after we bought MS stuff.

>asked if it would work with a win2003 server as opposed to a win2k

Our upgrade paths are fixed by MS. This absolutely sucks. Our systems require specific patch releases from MS. Once they stop supporting those patch paths, we have to upgrade. Agian, if you don't like it, write your congresscritter.

>but they didn't even know how to install windows

I'm throwing a bullshit flag on this play. I find it difficult to belive that no one knew how to install Windows. In the USAF, we have a NCC department that does nothing but install, configure, and maintain Win2k servers.

There may have been an internal power play based on getting Win2k3 server training. That is an ongoing military issue. Your boss tells you to do something. If you do it and screw it up, they ask if you were trained to do that thing. If you were not trained, then you go to federal-pound-me-in-the-ass prison for working on something without proper training. If you were trained and you screw it up, then you get in trouble for not folowing the training guidelines for whatever it was you broke.

Everyone working in a military NCC can install Win2k Workstation and Server. Many of them are MSCEs or higher. They could probably install Win2k3. They just wanted official training on that product before they tried something and broke it.

>Installing a a windows server is a mountain of a task for them.

No it isn't.

>Installing .NET is something that, as they say, they 'have been working on for

Re:Brief Rebuttal

[Osty]

This should have been in your install documentation anyway. Every contracted program I have ever dealt with has included install documentation. Most of it also includes referneces on how to install any required software. If your documentation needed revising, then admid to it. If they just needed to read the documantation, then tell them that. Don't just give someone a CD with a database and demand your paycheck.

Installation documentation is for the lazy. They should've had a proper MSI package that would install the database directly, without any intervention (well, maybe a gui on the installer asking a few easy-to-understand questions, or a few documented properties on the installer command line). Handing over a dump of a database, with or without documentation is weak, especially since it's hardly difficult to write an installer. Especially if you're smart about your database deployment (ie, consider detaching a clean copy of your development database, packaging up the .?df files, and re-attaching it to the production server via an MSI, rather than a shitload of files containing the schema that needs to be loaded).

Re:Brief Rebuttal

[vladkrupin]

Oh, brother... feel the need to reflect on what you said, but don't even know where to start.

Sorry if I offended you, but don't take it personally. Besides, we were not working with USAF, in fact, not with the military at all, so most of my statements probably do not apply to you by a long shot.

The thing I was really mad about was that they hire someone without any knowledge whatsoever to do their IT. When I was saying that installing windows was a mountain of a task, I meant it. Not kidding. For that particular person it was. For you it isn't, that's cool, and I have no problem with the government employing you to do that stuff. I have a problem with them employing someone with the computer knowledge and experience that is comparable to my grandma's (sorry, grandma!), and me having to pay for that.

>Installing .NET is something that, as they say, they 'have been working on for a while, but haven't got it quite yet'!

Probably awaiting training for .NET stuff.

Well, I have no problem with their selection of the platform and choice. It wouldn't be mu first choice, but that's cool - different tools are for different jobs, yada, yada... The problem I do have is that extremely stringent requirements are made, which, in turn make us turn out more expensive code, which, in turn we all pay for. And it turns out that the requirements did not have to be that stringent at all - not only they do not have that infrastructure in place yet, but they don't even know why/if they need it. Quite contrary to what we've been told all along till the moment came to deliver the project.

I agree, but YOU give us the money to run our systems.

Not really, if you think about it. Yes, I do decide to live in the US, and hence I indirectly decide to abide by the US laws, including paying taxes. On the other hand, I have rather limited chances to actually decide how my taxes get spent. Yes, thanks for pointing out that I can talk to my congressman. That's about the only thing I can do, and I haven't really seen much come out of it :( Not that I really expected to either. So, while, yes, I give you the money to run the systems, I don't really get to chose whether YOU run them or that other guy with an apparent total absence of any computer experience.

The American people have decided that my CCNA/CCNP, 5 years of linux experience, 8 years of Windows experience, certifications in ATM, HP Openview, Promina, etc... are worth $32k/year. I don't bitch about my paycheck.

There are two things I have to say about that.
1. That sucks.
2. I probably should take a few lessons from you on how to be content.

But seriously, this is aglaring mis-management of resources by the government in general, just as I said. Firstly, they underpay you, and thus reduce your motivation to work, and also risk loosing you - a valuable asset with tons of training, knoledge, etc. Secondly, they hire a guy who is not qualified to do the job, but nobody really knows how much he knows because they don't have an HR person who can tell a difference. The resume looked good, so he got the job. Probably for a similar salary too. Both cases are a gross mismanagement of resources entrusted to the government by the taxpayers. I would love to see that guy getting fired and someone like you getting an extra, say, 32K to your paycheck. I bet you wouldn't mind either for just a few hours a week (I doubt you need to spend any more to do that guy's job). Too bad I don't get a say in that matter... For that matter, don't invade Iran or whatever the next US target will be and add it to people's paychecks. I'd applaud this decision and even grumble less about my taxes. Too bad I have no say in that either...


If you don't like what you saw, you can either fire us or try and come up with a plan to fix the problems you saw. Either way, we both win.
No, I don't like what I saw. Not a bit. No, I am not going

It's 'Unit Tests', stupid!

[adamscottphotos]

I'm amazed that I haven't seen a single poster bring this up yet so...

The key to quality software; flexible, extensible, fault-tolerant, maintainable, and all of those other adjectives that 'good' software is supposed to have is very, very simple.
It's called Unit Testing. It's not brain surgery. I have worked on several medium to large scale projects (500k-3.5m lines) in several languages and environments, and I've yet to bomb one hard using this methodology, despite the usual client shenanigans.

Every time I write a functional subunit, I start by writing a series of tests (based on the spec, hopefully) that define 'doneness' for that subunit. Every object in the system has it's own set of tests. The test harnesses are chained together, so I can hit a button, so to speak, and run all of the hundreds to thousands of tests at once.

Whenever I check in new code changes, I run the test suite. If a test fails that previously worked, then I broke something. This plus good OOP practices (low coupling, high cohesion) allows you to make changes on the fly without the kind of 'The Money Pit' syndrome (fix one thing, another breaks) that is described in the article.

I am certain that the system in question was NOT developed with these methods. Most development organizations that I come into contact with pay lip service to the concept, but don't want to spent the perceived extra $ up front. The thought of all those developers writing TESTS when they could be writing CODE scares the willies out of them. But it pays for itself. It really does, every single time. Don't tell your boss, and try it on your next project. This is old news - google has a ton of info on it, and there are some good but unnecessary books also.

In the dot.com glory days, we had a huge system, running several hundred transasctions per second on a geographically distributed system of clients. We made fundamental architectural changes without a hitch, switched servers live without a hitch etc, and made a zillion little changes, all live, and all without a hitch (well, other than really stupid human errors, like locking out the client upgrade system with a bad password... oops). We had zero budget, and 2.5 developers. Unit tests are the Way, and if any company that doesn't mention them in your first meeting, run like hell.

Re:It's 'Unit Tests', stupid!

[BenjyD]

Automated unit testing is definitely an excellent way to ensure broken code doesn't get out the door. How code gets released when the only guarantee of it working is that a developer 'thinks it'll work' is beyond me.

Reactive, Just-In-Time Development

[prestidigital]

I've read article and many of the replies here. Several /.ers definitely describe flaws in government contracting processes and hiring practices that I've experienced, but I think they are missing the point. I think there is an additional, more fundamental flaw, that has been overlooked - or at least didn't get modded up high enough for me to see - maybe i should go trolling for a different set of opinions :^).

My experience tells me that the problems begin when we fall into the trap of trying to solve problems with a reactive mindset instead of a proactive mindset (proactive being favorable). We allow daunting problems and/or a need for revenue to back us into a corner time and time again and every time we are forced to hack our way out. Some of that is just old-fashioned survival, but a lot of it can be avoided with deliberate forethought, planning, discipline, and a commitment to quality and detail.

Avoiding clusterf*x has to be an institutional effort, whether the institution is a huge goverment agency or an tee-tiny, independent software shop. Everyone in the organization - operations, sales, IT - has to be on board with the policy that "if it's worth doing at all, then it's worth taking the time to do it right...the first time." I said "fundamental" earlier b/c that has to be something like lesson #5 in life's little handbook - we all heard it all too often when we were kids, we know it's true, and yet now we don't pay it any mind.

I do think the failure to heed that simple maxim usually starts in business development and snowballs by the time it gets to IT, but it really goes both ways. Everyone has to be responsible for maintaining the discipline required to produce quality.

What happened with this system is everyone involved got themselves all in a panic like a drowners who not only won't let you save them, but pull you under too. It's understandable given Sept 11, but "undertandable" and "right" are two different things. Legislators threw money at a situation they didn't try to understand. Deal-makers when after that money, promising to solve problems they knew they didn't understand. Developers enabled deal-makers by claiming to understand. No one took the time to do it right the first time.

P.S. It doesn't have to be this way.

It's not just one problem to solve

[Anonymous Coward]

Here's a quick bulleted list of items that I see as causing problems with the government projects I've been involved in.(yes, I've been doing reports for management waaaay toooo muuuch )

1. Requirements change over time, but inadequate funding available to really redo the design
2. The software was developed and designed for a 4 year life cycle and unfortunately we are in our 5th/10th/20th year of maintenance
3. Old software is compared to brand new commercial apps and their features, which didn't exist when the software was originally written. (Think of comparing a telnet, ASCII based application to a GUI app. And yes, ASCII based systems do still exist and are used for real work)
4. Use of cheap, inexerienced people in order to make a higher profit margin
5. Unable to charge a per usage/license/maintenance fee, only able to charge for the development costs up front, so potential profit avenues are reduced.
6. No integration planning on the government's side due to lack of expertise, politics or not realizing that different projects should be coordianted among contractors.
7. Eye Candy tools sold to higher ups without adequate technical input into the chosen packages.
8. "We've never done it that way before" uttered by company management, customers, developers, etc, which leads to long involved discussions and training basically as to why new Method A might be better than old Method B.

Now, the really, really hard part, what to do about it? I dunno, if I knew that, I'd be getting paid a lot more money then I am now and I have my underlings read /. to me while I sat by the pool. I am trying to do stuff, as I can, so that I can feel better about the projects I work on.

Myself, I am working on educating people in our company about other ways and means to make money from software and trying to insidiously ;-) get more competent people on the project in order to recover from past errors that are really biting us in an um, bad place. Progress is being made, but man, it is slow. (Think glaciers and you'll get an idea.)

You guys have only a vauge idea

[mess31173]

You guys have only a vague idea of how government works. Or if you do know, you have had vastly different experiences than I. I worked for the Dept of Revenue and Finance for a state government and this is exactly how things worked. Each government body, Natural Resources, Transportation, ect, ect each got a set budget based on how much they spent the year before and how much money the senate/governor allotted them from the total state budget that year. The big deal in our agency was to spend ALL of the money that was allotted to us by the end of the fiscal year otherwise next year we would lose what we didn't spend and our budget for next year would be shrank to the same ratio as what we spent the year before. Our budget would actually decrease if we didn't spend the all of the money in our budget!! Logic would tell me that if you can justify your budget but can manage it well and run UNDER your budget you would be REWARDED not the other way around. I digress. So, anyway, at the end of the year the director of the project would tell us to make a "wish list" (he actually called it that) and we got to buy pretty much anything we wanted, including contractors, and do anything we wanted. So, need a PDA? You got it, a new workstation? Got it. You name it, as long as we didn't deplete the pot we got it. Kind of crazy how the system encourages waste like that but it does.

I just live here....

FUD conspiracy theory

[Stephen Samuel]

Hey: If the government wants to put the fear of .gov into foreign students, then what better way to do it than this? Laws that mandate that students have no freedoms if they're not in a database and a database that randomly scrambles their data.

The result is that every once in a while a random student is arrested, delayed and/or deported because of a glitch in the database. As was mentioned in the article, students weren't too worried before SEVIS came into service, but now, the government is acting like a rampaging despot -- but they get to blame it on an opaque, mallfunctioning computer system. What more could they ask for?

Now, if the government wants to deport a student for (say) calling Bush a moron, all they have to do is induce a glitch in SEVIS and then have the student stopped for (say) speeding. One quick look in sevis, and they're in jail or on their way out of the country.

This whole randomness aspect is what has the foreign student body nervous, and it's almost impossible to pin down. It seems like a pretty good opposition supression system.

. . . But as a corrolary to Occam's razor says: "Never blame on belligerence what can be explained by simple stupidity." -- and we're talking about the government, after all.

Look at is this way...

[tundog]

Maybe the bugs where put there by a malicious programmer (slashdotter?) who disagrees with all this tracking nonesense. No better way to sow the seeds of discontent.

I was talking about something similar to a colleague about how I wouldn't be a part of a research proposal for the DoD because I fundamentally disagree with concept of a Big Brother State. He said that he actually new a researcher who had taken on defense-funded research projects in the past and just tanked-it on deliverables because of his own ideological sentiment. Not only was the research rendered innocuous, but it tied up government funds that could have been used on other more incideous projects.

Not an approach I would champion, but interesting nonetheless...

DOI Websites Yanked *Again* for Security Flaws

[mattOzan]

Remember in late 2001 when the US Department of Interior was ordered by the court to take more than 100 of their web servers offline due to abysmal security? Hired white hats were easily able to gain access to the US Indian Trust database and found no security measures or even audit trails in place. Worried that this could be contributing to the agency's continuing mismanagement and loss of allegedly billions of dollars belonging to Native Americans, Judge Royce C. Lamberth ordered the DOI to "immediately shut down Internet access from any computer, server and system in the department that has access to individual Indian trust data."

The defense counsel noted that the fact that they took down over 100 mostly unrelated servers "...just shows you how inept they are. They don't even understand how these systems relate to each other so they just pull the plug on the entire system."

And now last month they were ordered to disconnect their servers again after refusing to let a court-appointed special master test the security measures they've supposedly put into place since then.

Sounds like an endemic problem for government agencies, at least at the federal level.

Don't worry, screw-ups are not just for government

[NotQuiteReal]

Having worked for some large companies (directly or contractually) I have seen plenty of IT disasters in private industry.

The common thread is typically not an engineering failure, but rather, egos, politics, and ill defined requirements.

Like many things in life, there is usually no big conspiracy to screw things up, they just get that way "one good idea" at a time (e.g. The road to hell is paved with good intentions).

XML-enabled Kafka

[Animats]

The basic idea is reasonable, for a Big Brother system. But apparently EDS (Ross Perot's old company) botched the implementation.

SEVIS, like most web-oriented applications, consists of a browser form based interface, application logic, and a database. There's also an XML-based batch update capability. It's all reasonably modern and buzzword compliant.

Looking at the manual, the main problem you would expect to see is rejected transactions. Basically, the system won't accept transactions which indicate something that violates immigration policies. That makes it inherently obnoxious.

But on top of this, apparently the database back end has problems too, since data is reportedly "bleeding" from one school to another. That's not something that an end user (a Designated School Official) can cause; school officials don't have broad enough access rights.

Any major error in the data for an individual tends to prevent transactions which could correct the error. Designated School Officials can only enter transactions which are consistent with the existing data. Anything else requires approval from what used to be the INS, and is now part of Homeland Security. So that creates a huge load on an organization that's always been behind on paperwork.

All entry points to the US can access some of this data, and it's used to decide who gets in, who gets turned away, and who gets sent to a detention center.

The whole system is very Russian - great power over individuals, exercised ineptly.

"You want a vision of the future, Winston? Imagine a boot stepping on a face for eternity" - Orwell.

local governments are mostly co-opted now

[zogger]

--there are very few "pure" local governments in the US any longer, and the REASON is that the big federal dollar carrot. Our federal tax rate is extremely high, and most local governments are forced to accept BOTH federal money back to them,after first having a significant portion skimmed for the federal bureaucracy, and always with strings attached "do it this way or no money". that's control, which means the feds control everything. For example, several states have voter referndums, and have voted to allow medical marijuana, but the feds say no, so it's still "illegal". It's pervasive, and crosses into all avenues of governments, from roads to schooling to you-name-it. It's even gotten down to miniscule and obviously supposed to be local zoning, you'll find a lot of it follows federal guidelines now, and most will be changing over. We have a theoretical Union made up of soverign States and the districts and territories,etc as you outlined, but because of how they are funded,and a plethora of federal laws, regulations, edicts, findings, orders and directives, they all (mostly, conversationally/generally speaking now) are just agents of the feds. I think you'd have a hard time finding many local government exceptions to that, state or county. It's an illusion, just like born-with constitutional "rights", independent judiciaries, the jury system acting as a check on bad laws, "real" specie backed money, and free and honest elections.

Re:Nobody read this

[surprise_audit]

From the article:

EDS built the pilot student-tracking system after the 1996 immigration law was enacted. The company's current work on SEVIS is an outgrowth of that project...

So, apparently, SEVIS is inflated from the original (1996) prototype.

Actually, I don't see your "mid-late 80's" quote in the article. I do see "late 1990's" mentioned just under the Born Too Soon header, going on to the previous poster's quote and the above reference to EDS' pilot project.

Ever try bidding on a government contract?

[putaro]

The reason the same gang of idiots keep getting work from the government is that they're the only people willing to bid on it. Government procurement is so bizarre that unless you have a team of specialists putting the bid proposal together you have no chance of getting it. Every large company I've worked at had a special "Federal Systems" division whose job it was to deal with that sillines.

Too much job security?

[BenjyD]

The problem with most of these projects seems to be the lack of that standard incentive scheme most companies run which goes:

Do job really badly -> lose job

Here in the UK, government IT projects go vastly over budget and fall apart time after time, yet the same companies get hired again and again to do the contracting.

Low-bid may not be the problem ...

[OldHawk777]

Well ya-know,

Low-bid may not be the problem, competent selection of appropriate technology products and capable competent vendors as a low priority to cost may be the problem.
I mean, technologist (Tech-Advisors) are not the government employees on the selection/review panels. They are Legal, Contracting, Acquisition, Logistics, ... specialist with little or no idea of commercial market technology application/architecture requirements.

As a matter of public policy (and taxpayer money) is this level of non-functionality to be expected in these sorts of projects? No, I suspect (no proof), as above ....
Is the contractor just ripping off the taxpayers with bad code? Yes, I suspect (no proof), as above ....
How hard is it to write software like this that works? Not very hard ..., I suspect (no proof), as above .... Well supposedly if it could not be done correctly, competently, successfully, ... then the OEM/OSD vendor and government contracting folks should state in the R&D SOW requirements (or someplace in the signed contract) potential problems and possible failures to meet expectations. Everyone involved are supposed to know what they are signing, the requirements, and limitation. So, if you don't get what you paid for, then whose fault is it in a court of contract law.
The article focuses on the SEVIS database, but have others noticed similar trend in other government information systems? Yes/No I suspect (no proof), as above ....

REMEMBER: It appears the game is to maintain government management levels, while reducing worker-bees and pack-mules (contract out wherever and whenever possible). This will simplify personnel management, provide vendors to workload and blame when problems and cost escalate, maintain the military industrial complex, and funnel more tax dollars into the growing economy at an appropriate level ... just like the trickle-down tax-cuts are already helping US.

OldHawk777

Reality is a self-induced hallucination.

So, never take what anyone says very seriously ..., Okay.

Government work environment

[Anonymous Coward]

I've been a programmer for the government for 7 years now, 5 of that has been at one agency. There's several problems that arise in a government programming shop, many of which are beyond the control of the programmers themselves reguardless of proficiency or work ethic.

We can't really use the best tools for the job, we use the tools that someone on high has mandated as the agency standard. I've been in more than one situation where we're using the wrong tool for the job, but we have no choice. Also, on the rare occasion that new and/or appropriate technology is used for a project, training to use that technology is usually given after the project is due to release. Management pounds out "ambitious" deadlines to try and win contracts, which obviously bites their programming staff in the ass. I recently worked on a project that had about 5 months of design development and test time, the developers caught wind of it 2 months into the schedule, and we didn't get all the requirements until 3 months in, which left us 2 months of actual design programming and test. We had no one with experience in any of the tools, technology or language, not even the project lead. Fortunately we had a great team. I strongly believe that any skilled set of commercial developers would have had a hard time in the same situation.

Not all government programmers are inept, and those that are inept are impossibly difficult to get rid of. There are very intelligent, self teaching, skilled programmers in my shop, and for every one of those programmers I can point out 3 that need their hands held while they write their code. The pay in the government sector doesn't attact many highly skilled individuals. The skilled people in this sector either get skilled after they've been hired, are in dire need of work (which means they'll leave as soon as something better comes), or they feel a patriotic duty. I have a sneaking suspicion we don't have a lot of highly skilled and patriotic tech geeks. Anyways, as I was saying, those inept folks are hard to get rid of. You have to have piles of paperwork and documentation proving that you've tried to rectify the situation, and then the person working on cutting the fat usually comes under fire by the union. Essentially the boss gets skylined and comes under intense scrutiny by the union to decide if her/she is just "gunning" for the poor employee. It's easier and less stressful to just try and get another shop to take the useless individual. We've got a couple hot potatoes in our shop, and unfortunately we were the last group to discover that they are useless, which means we get to keep them.

If you want things to change, get involved, take a healthy pay cut, work under stifling beaurocracy with inept slackers and make a difference!

EDS not Technology failure

[xdonnel]

Cost is rarely the only criteria on large public sector contracts
With EDS's track record of failure and their financial instability,
it is hard to understand why they still receive these contracts.

This application could be developed successfully on any modern platform

EDS is most likely responsible for most issues raised in the article if they were the main contractor.

The "bleeding" issue is the most serious issue raised.
If the system cannot maintain data integrity, then it is worthless for
the main requirement of monitoring students who may be a security threat.

The "bleeding" issue is probably caused by
failure in database design (EDS resp)
or less likely a serious performance issue

"to create a record for the newborn child.. ..delete everyone's file and create new ones"
failure in database design (EDS resp)

"Before extra capacity was added to the system, several student advisers
logging on at once caused SEVIS to slow dramatically"
Flawed operational architecture even if sizing, availability requirements were poorly defined as solution did not scale (EDS resp)

"help desk for long periods, only to be offered a solution that turned out to be wrong"
Failure possibly due to organization, staffing level, poor documentation, inadequate handover to maintaining organization to provide support at the required SLA (EDS resp)

"..several others, she says. "It begs the question,
how rickety is [SEVIS] if you can't do upgrades to it?"
poor maintenance procedures, no regression testing (EDS resp if maintaining)

"The Bureau of Immigration and Customs Enforcement
has yet to name an official to oversee SEVIS."
Inadequate handover to maintenance organization (joint failure)