Slashdot Mirror
Glitches in Massive Government Databases?
HBergeron asks: "Rather then post this as another YRO in the litany of new government datamarts there is a more fundamental question for all the coding Slashdot readers out there. This story, in Government Executive magazine, outlines the range of programming glitches in what is a relatively simple database. As a matter of public policy (and taxpayer money) is this level of non-functionality to be expected in these sorts of projects? Is the contractor just ripping off the taxpayers with bad code? How hard is it to write software like this that works?" The article focuses on the SEVIS database, but have others noticed similar trend in other government information systems?
...Except perhaps to the executives the magazine is aimed at. Early versions of software are generally pretty buggy, particularly if the target keeps changing, and most especially if it is in response to a hastily crafted law. The only thing that's surprising about this is that the output is taken so seriously by law enforcement officials *prior to completion*.
Don't they have some donuts that need eating?
When it comes to government, failure is rewarded with more money. In fact, failure is often cited as proof not enough money is being spent.
Besides, why bother doing a good job if you know you'll get paid either way? That's what the tax collector is there for!
Look no further than the fiasco with Canada's National Gun Registry to see how badly the government can screw up a database project.
After living through that I no longer have any confidence at all in the government to be able to implement any IT project competently. One billion (with a B) dollars to develop a database that essentially matches a gun serial number to a name and address. And they're still not done. HOW? Somebody please explain that one to me...
After September 11, it was decided that the system had to be used starting early 2003. This was years earlier than in the original plans.
I have no experience whatsoever with SEVIS, but here a few tips I can think of on the fly:
-Xerox'd copies of any forms you've filled out related to all this. Carry these with you on the plane in carry-on luggage.
-Ask your advisor for a hard-copy listing of all the data that will be entered into the SEVIS system related to you.
-Consider having a letter written and signed by the advisor that entered your data into SEVIS, indicating what he/she has done. Also consider getting that advisor's office and home phone numbers in the event something goes wrong outside the 9-5 timeframe. Again, carry-on it.
-Call your advisor from Australia and ask that he/she check to ensure that your data is in the system before you leave.
There are probably a bunch of other things you can do, but the point I'm getting at is that you should try to cover all your bases and double check everything. Yeah, it's unfortunate you have to do this s---, but it might be the only way to prevent yourself from too much hassle.
The One Rule Of Chess You'll Ever Need: Don't play someone who carries a kit in their bookbag.
I disagree with you. In my experience, every interface I have written to an EDS system has had to work around collosal conceptual errors on their part. YMMV.
"God is dead." - Frederik Nietzsche
The nuclear materials database was found to suffer from flaws in Microsoft's SQL server back in 2001. This series of flaws led to the amount of tracked materials being incorrectly reported although the actual information was being tracked correctly. The origional slashdot article can be found Here.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
I spent time in Uncle Sam's Misguided Children, so yeah I knew there is/was fudging. Also [personal experience and no I was not in S4, logistics and supply] if a unit/department did not spend its entire allocated budget by the end of the fiscal year said budget was reduced. There were massive buying sprees of barracks supplies, spare parts, office furniture, etc. While not wasteful, unless it had a short shelf life, we wound up buying all kinds of crap that sat around for months before it was used. I'm not disagreeing with you; I'm just adding another expense theory.
-- Some days you're the dog; some days you're the hydrant.
All too often, what happens is that someone either sits down and knocks out a database out of thin air (with no understanding of the underlying business needs), or a team sits down with the latest and greatest methodology and tool set and forces the process into their pet methodology and tool set (again, with no understanding of the underlying business needs). I don't think perceptive /.ers need to be told what happens next.
But why does this happen? IMHO, we have dismissed the value of generalists, those who understand the underlying business model, the people and processes involved and the technical means available. At best, most teams have a business expert, a development expert, a database expert and a project manager, who likely does not have business expertise and, hence, cannot act as a proper liaison between the business and technical sides.
The generalist, being neither fish now fowl, and being more mature/older to boot, tends to be the target of early retirement and layoffs. And yet, these are the very people most needed by both public and private sectors. They are the ones who can make sense of the customer spec and present it to the team in a fashion that makes sense. They are the ones who can explain development realities to the business side in language that they can understand.
Does this mske sense, or is it the nostalgic longing of a generalist?
I can think of several widely used systems that are backed by databases that work just fine. Slashdot wasn't built on a massive budget. Amazon doesn't have a history of "bleeding" data from one user to another. Google and Yahoo are certainly capable of handling tremendous loads.
I see several possible problems here. First, it is possible that this software was rushed into use before it was ready. Given the political pressures involved, I suspect that is part of the problem.
Second, I doubt that all of the programmers involved are of guru caliber. I don't intend to malign them. Even assuming that you have nothing but above-average programmers, when you build a huge project with lots of designers and coders, there are going to be miscommunications and some details that just aren't communicated.
Third, I would bet that this project has so much design documentation done up front that it is impossible for anyone to wrap their brains around the whole thing. This is, at best, a 1.0 release. And there are going to be design flaws in it. And the guys writing the code aren't likely to have a broad enough overview of it to spot them all. They also undoubtedly tripped over a lot of things that weren't specified up front and should have been. It is the nature of the game. But they weren't free to just choose a good solution when the questions came up.
The projects I cited at the beginning were developed by small teams with a vision of what they wanted to build. Within the constraints of the tools they had to work with and the general idea of what they were building, they were free to change the rules. They could refactor to their hearts' content. That is not going to be the case on huge government contracts.
Everything that we know about open source, agile/extreme programming, etc. doesn't apply to this kind of project.
The net will not be what we demand, but what we make it. Build it well.
"They can't keep track of paroled felons, the database of people who can't vote is horribly flawed,"
Then don't live in Florida. It was a system designed to try to help Florida local governments enforce a Florida state law. This has absolutely nothing to do with the federal government, which is what the federal constitution mandates.
The way you genericly complain about "the government" shows one of the problems in these information systems: a total lack of knowledge of how "the government" works here in the US. We don't have one monolithic government (yet), we have a federal government, 56 state/commonwealth/district/territorial governments, and 3,066 county/city/parish/borough governments. Each of them have different rights and responsibilities spelled out by constitutions and charters. Each of them operates in a slightly different manner from all the others to reflect the wants and needs of its people.
I'm reminded of a story here on Slashdot a few years back about how France's government went "online," giving internet users the ability to do things like get a driver's license or enroll in public schools on a website. There were a great many people who complained about how the US hasn't done anything like this, all by people who didn't seem to realize that just about everything this new French site did are the responsibilities of state and local governments here in the US.
The only way things like this will get easier is if the US shifts away from the federal republic model and become a monolithic republic ala France. This would involve disemboweling the federal constitution and burning all state constitutions. Personally, I'd rather not see that happen.
Not this again. Once again, the reason for the $500 hammers, etc., is a matter of cost accounting, not nefarious fraud. These "scandals" erupted because the cost accounting used for a number of government contracts spread the large overhead and R&D costs equally by product line item. In other words, if there were 1,000,000 items being purchased, and $500,000,000 in R&D costs, each item was assigned $500 in R&D cost, whether it was a $50,000 custom circuit board or a $10 hammer. A lousy way to do cost allocation? You betcha. Fraud? Nope. See this for more info.
While this was ment as a joke, i'd like to point out that people valuing time is the buisness model for all commercial linux vendors. You can get Red Hat free, but many companies pay lots of money for support.
I've been a programmer for the government for 7 years now, 5 of that has been at one agency. There's several problems that arise in a government programming shop, many of which are beyond the control of the programmers themselves reguardless of proficiency or work ethic.
We can't really use the best tools for the job, we use the tools that someone on high has mandated as the agency standard. I've been in more than one situation where we're using the wrong tool for the job, but we have no choice. Also, on the rare occasion that new and/or appropriate technology is used for a project, training to use that technology is usually given after the project is due to release. Management pounds out "ambitious" deadlines to try and win contracts, which obviously bites their programming staff in the ass. I recently worked on a project that had about 5 months of design development and test time, the developers caught wind of it 2 months into the schedule, and we didn't get all the requirements until 3 months in, which left us 2 months of actual design programming and test. We had no one with experience in any of the tools, technology or language, not even the project lead. Fortunately we had a great team. I strongly believe that any skilled set of commercial developers would have had a hard time in the same situation.
Not all government programmers are inept, and those that are inept are impossibly difficult to get rid of. There are very intelligent, self teaching, skilled programmers in my shop, and for every one of those programmers I can point out 3 that need their hands held while they write their code. The pay in the government sector doesn't attact many highly skilled individuals. The skilled people in this sector either get skilled after they've been hired, are in dire need of work (which means they'll leave as soon as something better comes), or they feel a patriotic duty. I have a sneaking suspicion we don't have a lot of highly skilled and patriotic tech geeks. Anyways, as I was saying, those inept folks are hard to get rid of. You have to have piles of paperwork and documentation proving that you've tried to rectify the situation, and then the person working on cutting the fat usually comes under fire by the union. Essentially the boss gets skylined and comes under intense scrutiny by the union to decide if her/she is just "gunning" for the poor employee. It's easier and less stressful to just try and get another shop to take the useless individual. We've got a couple hot potatoes in our shop, and unfortunately we were the last group to discover that they are useless, which means we get to keep them.
If you want things to change, get involved, take a healthy pay cut, work under stifling beaurocracy with inept slackers and make a difference!