Slashdot Mirror
Glitches in Massive Government Databases?
HBergeron asks: "Rather then post this as another YRO in the litany of new government datamarts there is a more fundamental question for all the coding Slashdot readers out there. This story, in Government Executive magazine, outlines the range of programming glitches in what is a relatively simple database. As a matter of public policy (and taxpayer money) is this level of non-functionality to be expected in these sorts of projects? Is the contractor just ripping off the taxpayers with bad code? How hard is it to write software like this that works?" The article focuses on the SEVIS database, but have others noticed similar trend in other government information systems?
I think this movie shows what a *real* totalitarian state would look like: the danger to our freedoms is not from corruption but from incompetence. Programs like TIA creep me out because I'm absolutely certain that the Feds will find a way to fuck it up and throw some poor guy in detention because the computer skipped a byte and came up with his name. Ditto for the PATRIOT Act. Few people have recognized this, but what's frightening about Ashcroft is not that he's a fundamentalist autocrat, but that he's an incompetent fool. If innocent people suffer from the government's extension of powers, it won't be due to the GOP taking out its enemies but because some FBI secretary got a virus on her computer.
I'm not a libertarian; the government indirectly pays a large portion of my salary. However, the extension of government power worries me, because the more control they have, the more opportunities to fuck our lives up.
Fixing broken EDS code is a large part of my job, the SEVIS project is no doubt another example of EDS shoddiness. The EDS business model seems to be as follows: - Collect $200/hr from client. - Pay h1-B $15/ hr to produce complete choss. - Management keeps the other $185/hr. for second vacation homes etc. But I suppose it is better that this project fail, at least we can count on EDS for something. MM
This is the same old software engineering problem, over and over again.
A user who has never written a *COMPLETE* system specification, acutal has no idea what that is, who only knows what he/she does not want.
Software developers/coders/bodies who are not SME's (subject matter experts), making system / software decisions without either the knowledge or guidance to understand the ramifications of those decisions.
Neither users, nor software development companies want to deal with these issues, they would rather just get the money.
That is why most large software development/ service companies have such bad reputations.
According to SEI, (Software Engineering Insitute) over 70 per cent of all software development projects are terminated as failures.
The quote from the original article:
.NET 2003 server with M$ SQL, etc., etc. We did our best to make them consider PHP and the like, because that's what we normally use, but they were willing to pay extra to have that stuff developed in all-M$ stuff! We were told that the reason for that was because their IT was managing only M$ software, and the server was already there, and they couldn't have anything else (e.g. PHP). Fine, I can understand some bureaucracy in IT - that's cool, but imagine my surprise when, after we shipped them a CD with the project, they called us back and asked if it would work with a win2003 server as opposed to a win2k!!! Not only they didn't have the server yet, (or the infrastructure for that matter), but they didn't even know how to install windows! Which brings me to point #4:
.NET is something that, as they say, they 'have been working on for a while, but haven't got it quite yet'! And, when we give them a database dump they have no idea what to do with it and you have to walk them through the process (right-click on the 'Databases', select 'Create New database', click ok...) And they are paying these people!!! Errr... Let me re-phrase that - We are paying the government to employ those dumbheads! Thatnks goodness the network on which that is installed is not connected to the internet - the same idiots are in charge of security as well.
How hard is it to write software like this that works?
Wow! Well said! My grandma couldn't have done better. In other words, please define 'works' for me. How many blue screens a day constitute 'works' and how many are too many?
Also, since we are at it, I want to reflect back on the latest project we have done. Incidentally, for the government. Before asking if a vendor is ripping the taxpayer off we need to consider how the government mismanages the resources it has. Consider the facts:
1. The project itself was fairly small and simple. I'd say it would normally take about 2 months to develop and deploy, but it needed to be done before the end of the fiscal year, so it was a 'now-or-never' situation, and was a horrible time-crunch. We had slightly more than half the time necessary to do it, but they won't even try to install it till probably the end of the year! The quality of code would've been greately improved if we coded, say 40 hrs/week instead of pulling all-nighters.
2. They tried to keep tabs on the development by scheduling 'technical meetings' over the phone. While there is nothing wrong with that per se, in a time-crunch that was a horrible waste of time. The smartest things we've heard from them were questions like 'Are you using hungarian notation?' or 'is your code well-documented?'....
3. They insisted on
4. Their IT is kick-ass. As in 'their ass needs to be kicked real hard'. Installing a a windows server is a mountain of a task for them. Installing
Yes, it is true that some contractors will rip off the government (and it is really the government's responsibility to make sure that doesn't happen! But that's not the point). The point is that even if they have a perfectly good product developed by honest people, they are still remarkably talented at screwing it up. Bureaucracy and lots of idiots in charge of hiring people are to blame.
Jobs? Which jobs?
> And the government system of going with the lowest bidder is bound to cause some problems
I worked in a state agency, and the fact that we were required to take bids didn't really change who we purchased from. We just chose the vendor we liked best and justified it by writing the project needs around that vendor. They did that with employees too. When a new job opened, they often had someone in line for the position. However, equal opportunity required that the do interviews for the position. To justify the person they desired, they would write the job description and requirements based on the skills of the individual they liked. They would then schedule interviews even though they already had someone chosen for the position, just to meet requirements. I suppose they could have changed their mind if they found someone who was absolutely fabulous, but it's hard to convince an employer how great you are when in the back of their mind they don't think the interview is going to matter anyway.
Disclaimer : I have worked for a number of Financial Institutions and Large Corporations.
My experience with the problems of these sorts of situations is as follows:
1. Sales droids underbid each other to get the job and commit to ludicrous time frames
2. Project teams end up with short development time and are always pushing to reach the deadlines in time.
3. Client changes their requirements, but will not change their expected delivery date. Either they refuse due to business need, or they do view their change as an actual change. More often they view their change as a "clarification" - even if it contradicts what their specifications orignially said.
4. Agressive job market has Project Managers kowtowing to Client demands.
5. Multiple departments are clients, but pay different amounts into pool. Each department seeks to maximise their benefit at the cost of other departments (despite fact part of same organisation - politics)
I mean, really, the problem exists in the fact business units will often not sit down and commit to producing clear, unambigious details of what they want & need. Bugs creep into the process when your dev's are working frantically to meet the deadlines and handle the unexpected change request.
And now a pithy little quote to put on your wall:
-----
Programming to Requirements is like walking on water.
It's easy to do when everything is frozen.
I've worked on some government contracts, and in my opinion a big part of the problem is in the GSA schedule rate structure that the Federal uses for contractors. It is much more profitable for a contractor on a government project to put many junior people on a project rather than a few senior people, for the same amount of revenue. For instance, a junior developer may cost a contractor $50/hr with overhead, but the contractor is able to bill the government for that junior developer at $150/hr., a spread of $100/hr. A senior developer may cost $100/hr with overhead, but can only be billed to the government at $175/hr, a spread of only $75. Furthermore, the contractor can bill more hours of junior time than senior time under a given budget cap, compounding the effects of the greater spread. Thus, the incentives for contractors are to use as many junior developers as possible on a project, to increase the profit margin.
Unfortunately, It's a rule of thumb in this industry that a few good programmers are a lot more productive than many unskilled ones. The result is that many government IT projects are shoddily built by well-meaning but inexperienced developers who are put in that position by a contracting structure that fails to recognize the realities of the IT industry. Contractors are just responding rationally to the incentives that are presented to them.
These numbers are examples -- in fact the situation may be even worse. Federal government contracts vary in their rate structures, and many are stingier than this. It may well be impossible to bring on a senior developer as a subcontractor because the maximum hourly rate that the government will pay on a project is lower than the cost of the senior developer.
A prime contractor that I worked with staffed a large WebObjects project for the Department of Defense with a dozen or so low-paid, fresh out of community college drones. Every six months -- when a project review was due -- they would bring us on board as subcontractors for six to eight weeks. In that time, two or three of us would take the code base from where it was four months ago and bring it close enough to the required progress to get the contract renewed, and then the prime contractor would say "goodbye" and toss us out. Four months or so would pass by, with their people making little meaningful progress, and we would get a panicked call for six or eight weeks of more work to get by the next project review. (Did I mention that the prime contractor didn't pay the bills for one set of work until they needed us for the next project review? It got so bad that at one point we had to treat them as though their credit rating was zero, and demanded that payment for each week's worth of work be deposited in an escrow account before we would continue.)
By the way, this rate structure is not unique to government IT projects. Other types of government projects display the same professional services rate structure. When I worked for a (then) Big Six accounting firm as an economist, most consulting projects for corporate clients were staffed with a ratio of one partner and two or three senior managers to six or eight associates. However, the Federal government group was staffed with a ratio of one partner and one senior manager to twenty or so associates. I talked to the senior manager, and he told me that (a) the associates in the government contracting group were paid much less than we were on the corporate side since they billed out at a lower rate, and (b) the only way they could make money was to use lots of cheap associates because senior people could only break even at best at government rates.
Ya know, it'd be nice to see a GSA person squirm over this sort of thing in front of Congress some time. Then again, Congress may be part of the problem, as they'd rather generate lots of jobs for constituents, instead of a few.
--Paul
I think that, for most managers, the analogy which would most make sense to them is that Programmers are pretty much the same as Lawyers:
The goal, in either case, is to take a set of rules -- often arcane and ancient (programming languages and operating systems Vs. rules and laws), and combine them in such a way as to allow the client to achieve their wanted ends.
The judge would be the rough equivalent of a wetware execution unit.
Once they accept the lawyer analogy, then you can ask just how reasonable it would be to expect a lawyer to accomplish a lawsuit according to a tight schedule. Although it is doable, the tighter the schedule, the higher the price (often exponentially so).
I came up with this analogy because I ended up, a few years ago, self-representing myself in a reasonably complex lawsuit (It was about 4 years old by the time I got pulled in). With a couple of months heavy research I was able to do well enough in the courtroom (before the chief justice, and later at the Court of Appeals level) to reasonably impress just about every lawyer I dealt with in court.
I achieved this by pretty much applying my programming experience almost one-to-one. I simply treated the legalese and rules of court as a programming language. Old precedents were treated much like code snippits.
If you look at old slashdot postings, I think you can see that good programmers don't have that tough a time with laws and even court decisions. I submit that it's because the paradigms aren't really that different.
Free Software: Like love, it grows best when given away.
Remember in late 2001 when the US Department of Interior was ordered by the court to take more than 100 of their web servers offline due to abysmal security? Hired white hats were easily able to gain access to the US Indian Trust database and found no security measures or even audit trails in place. Worried that this could be contributing to the agency's continuing mismanagement and loss of allegedly billions of dollars belonging to Native Americans, Judge Royce C. Lamberth ordered the DOI to "immediately shut down Internet access from any computer, server and system in the department that has access to individual Indian trust data."
The defense counsel noted that the fact that they took down over 100 mostly unrelated servers "...just shows you how inept they are. They don't even understand how these systems relate to each other so they just pull the plug on the entire system."
And now last month they were ordered to disconnect their servers again after refusing to let a court-appointed special master test the security measures they've supposedly put into place since then.
Sounds like an endemic problem for government agencies, at least at the federal level.