Slashdot Mirror
The FBI Software Upgrade That Wasn't
Davemania writes "Washington Post reports that the FBI's attempt to modernize their department has once again failed. The 170 million dollar Virtual case File system, the agency's second attempt to go paperless is reported to be useless. The finger seems to be pointing at the FBI leadership, greedy contractors and bad software management." From the article: "It appeared to work beautifully. Until Azmi, now the FBI's technology chief, asked about the error rate. Software problem reports, or SPRs, numbered in the hundreds, Azmi recalled in an interview. The problems were multiplying as engineers continued to run tests. Scores of basic functions had yet to be analyzed. 'A month before delivery, you don't have SPRs,' Azmi said. 'You're making things pretty. . . . You're changing colors.'"
If the anybody can screw up a big project like that it is the government. If it was 170 million of somebody's own money I think that it would have been done a lot better but since it is only the taxpayers money they seem to really mess things up. Perhaps this is one of the many reason we should limit the federal govt to their proper role as given in the Constitution.
Personally, I'd prefer the FBI not go paperless. Because (a) paper trails are nice in investigations and such (y'know, when the FBI finally goes up against the Supreme Court) and (b) stuff that doesn't have a hardcopy tends to get lost more often than physical objects...especially embarassing things...especially by government agencies.
Yes. I'm slightly paranoid.
Anyone else think the comments just weren't rendering right before they turned off ABP and saw ads?
Wouldn't it make sense to go for a more basic application as a first run, to at least provide a unified collaborative work environment, and use the working experience therein to define a more strategic, long term technology plan for the FBI? As I understand it, today's world involves many separate stores of information, electronic and not. Simply bringing those together in the crudest of fashions could provide significant gains in a relatively short time frame.
Stop by my site where I write about ERP systems & more
'A month before delivery,' Professor Knuth said looking up through his spectacles 'you can start implementing it if your correctness proofs are complete.'"
Ha! Welcome to the real world, guys.
They should have just started by picking a decent directory structure for the documents and then hooking up a decent search engine like the Google Appliance. Then the users could simply use web browsers instead of a weak, buggy, and expensive custom application.
Non CS people who commission custom software development often have no clue how expensive their ego driven non-standard features can be.
That quote was just too funny. In any huge huge huge project the state of play a month before is usually going through the thousands of trouble reports, deciding which features you will turn off so you can ge the release out, figuring out work arounds for the rest of them, discovering that the analysis was flawed on some functions and of course by now you ditched many other features because the analysis was not done at all. And yes you change a few icon colors to keep key users happy and get your sign-off. This is for a successful 1.0 implementation.
that's why government contractors that do this kinda shit (fail to deliver product) should be required to return all the money to the government, and if they don't they can rot in jail and the government will SEIZE their assets
If you cannot keep politics out of your moderation remove yourself from the Mod Lottery.. NOW!
Wow... I have never, ever seen a software product that wasn't working on QA bug reports right up to the minute the gold disc is burned. And afterwards, of course, working on all the pre-release bugs that had been classified as 'known issues'.
Chelloveck
I give up on debugging. From now on, SIGSEGV is a feature.
Sure, I think we all feel that way after reading this story. But the error could also lie with the Agency. If they are constantly asking for changes and new additions, what can the programmers do.
And this would make it very difficult to get companies to do government contracts in the future.
Perhaps they should have taken in past history as well as cheap price, when deciding on contractors.
You seem to have left out a step: The government agency changes the requirements after the bid is awarded, usually in the user interface. If you're a smart bidder, your low bid was only to cover the original specification... and any modifications are extra.
This might be a case of a contractor sucking at the government teat, but lets not forget that clueless PHBs and design by comittee can also run up the cost of a project without producing results.
A Human Right
I would like to discuss this in some detail. Congress Critters are you listening?
The issue here is simple and almost sinfully so. If you are to get a job working for the Government, it doesn't matter which agency, you have to provide your quailification credentials. We see them on looking for a job as a list of qualifications. This applies to contractors who supply the government as wee. This becomes a list that looks like a laundry list of the history of the agency. We programmer types will know these very well. The list goes like this:
Programmer with 15+ years Experience in Matlab.
Must have 15+ years in Military Logistics with US Army SMDC
and the list goes on. There are only two problems with this listing. Both of these cannot exist. The only person who can qualify with these "pseudo credentials" is somebody who has just retired from the army and frankly even then it is a fiction. The result is that old unqualifed failure that was just booted out of the ranks for incompetence is now the only person qualified for duty.
In the FBI this is worse not better. The results are that the FBI remains unable to respond because it cannot recruit new blood to infuse it. The situation in contracts is even worse since the bureaucrats in the Government reserve the right to pick and choose the people who will work with them on contract. The contractor doesn't get to pick his people! This makes absolutely sure that in the post 9/11/2001 world only those who failed us before 9/11/2001 can ever be "Qualified" to do any work. They are the "Experts" we hire to do our work. The resulting situation is from top to bottom the agency fails ever worse and costs ever more.
The solution is pretty simple. There needs to be a wholesale cutting from the top of the dead wood of the agencies. We need to fire about 99% of the people in management and start at the bottom rebuilding. The GS system is built the reverse of this. Bumping needs eliminated. GS people need to have manditory retirement at set ages and terms of service. The legislative support of contractors needs changed towards performance controls and away from managing personnel of contractors. Frankly the US agencies excepting where sensative data or methods are involved should have no influence over hiring of contractors. Even then it should only be security issues and not the qualifications at issue. Contractor companies should have to be performing based on results and paid accordingly.
Had the FBI contract had a penalty of $250,000 per day for failure to perform the results would be in hand and done now. This by the way is typical contracting rules in the civilian arena. I have worked on such civilian contract rules for years. It gets work done and on time.
As it sits it is typical for contracts to demand 10 years experience in .net. (Programmers will have to laugh their heads off on this one. .Net isn't that old itself)
Never Politically Correct ~ I prefer the facts If you don't like what I say, get a life, or comment yourself.
I heard a story a while back about a three-letter government agency who wanted a new air conditioning system put in. So the company doing it said, ok, I'll need to know how many people will be working in the building on average, etc., etc., and they were told that that's all classified, so they were forced to make a guess. Later, when the system didn't work so well, the same agency wanted to sue them, but it didn't get anywhere, due to the lack of fundamental information provided which was required for the optimal operation of the system in the first place. Typical.
pb Reply or e-mail; don't vaguely moderate.
I hate the lowest bidder system. It seems like the root of all screwups in the government. It's not as black and white as you are competeing for model number 00120 of product X. All but the simplest of cases shouldn't have to go through the whole lowest bidder system. Quality is extremely important and low bids don't take that into account. This story didn't really mention whether this was a low bid deal or not, keep in mind.
Look at pretty much any government building that was built on the lowest bidder system. I can pretty much guarantee it has mold or leaking issues.
If an officer ever threatens to taze you, say you have a pacemaker.
It would be nice if, sometimes, large organizations realized that applying computers to solving the problems of a paper trail is going to cause many many problems before any benefits are seen. In working with my university, I've seen time and again the tendency of higher-ups to see computers as a panacea to any/all problems an office might confront in keeping records on things.
For example, our housing lottery system was, until this past year, an in-person process where people were assigned times, showed up, claimed rooms, and was a fair system that worked. Then, the university got all fancy pants and replaced that lottery with this unbelievably crappy system called Residential Management System. To use: kill ad blocker, only use it in IE for Windows, ensure javascript settings are correct, and then wait until the clock allows you into the online lottery system. Attempt to use a non-intuitive UI that is completely new because you couldn't look at it before while time ticks away and other people claim the rooms you wanted. Even though I got the room I wanted, the experience was horrifyingly bad.
For these large organizations, I think less can be more. Keep your paper trail, but create a highly efficient system for digitizing documents. That way, you start to have some advantages of computers (search, organization, cross-referencing) without the liability of a completely paperless system. From here, you can slowly make a transition from leaning on paper to leaning on machines. But that would be the sane way of doing things, and we're talking about a governement organization here.
The new project is even worse than the old. No software, with the possible exception of truly safety-critical stuff like missle-control or nuclear power plants, needs to cost $425 million and take four years. You could have a custom OS written in pure assembly for a quarter of that!
Media that can be recorded and distributed can be recorded and distributed.
-kfg
We've learned this over and over again at my company. The likelihood of scrapping the whole thing because you've got nothing is logarithmic to the cost. That is, the more the costs go up, the more likely you scrap the whole thing.
The project has to be bitten in chunks. Lay out the functionality, and then start implementing it one small piece at a time, integrating as you go along. The Big Bang approach is always doomed to failure, or explosive costs, especially when you get to the reality that to deploy you need to shut down the business for two weeks to manage the data conversion. Lot's of small $1 million projects are more likely to succeed and be at budget then one big $20 million project.
This isn't news. It's the whole momentum behind a lot of modern development techniques such as Agile, or architectural such as SOA.
There's also a corrolary that any project involving a big consulting company like EDS, CSC, Anderson(or whatever the hell their name isnow), etc. is more than likely going to cost double what it should.
You misunderstand: allow me to translate:
They want a particular person. That's why the specificity. You need not apply.
If the specifications for the system were imprecise or constantly changing (as often happens), that would limit the ability of ANY software developer to create a stable functional system on time and within budget.
I'm not going to criticise the folks who were trying to implement the system until I know a lot more about the actual conditions in which they were trying to work...
Mainframe/UNIX Bit Twiddler and long time Windows/Linux Hobbyist.
The Theorem Theorem: If If, Then Then.
Oh, I see you've met my former boss. She was even worse.
her: have you completed X?
me: not yet, but I have Y and Z completed and tested.
her: but X is on the schedule to be complete today.
me: It made more sense to do Y and Z first, X will be trivial now that Y and Z are complete.
her: but X is on the schedule to be compete today. Y and Z are not scheduled to be done until next month. now we are behind schedule.
honestly, I can't make this stuff up. She actually said that as I stood there in slack-jawed amazement.
730K for $170 million? For the government, that is nothing. It is actually a pretty good price for that amount of code. It also seems like it was a pretty quick project too. And hundreds of SPRs a week before release? Not bad!
From my government contracting experience, none of this sounds that bad. Hopefully there is much more to the story that they aren't talking about. But from the examples they are using, SAIC's performance sounds distinctly "above average." It may have been a disaster, but TFA does not give us enough accurate information make that judgement.
I wouldn't be at all surprised if 90% of the functionality could not be provided by secure web servers and good quality wiki.
But that would be cheap and quick to implement and not much chance of making a vast profit.
Heck, give me half that--$85 million--and I'll develop the friggin' system myself.
/rant
You'd probably think so, but I bet after the first few months of totally contradictory change requests, specification creep, and an utter lack of hard-and-fast acceptance criteria, that you'd throw up your hands, too.
You can blame the contractors all you want, but I've worked on a bunch of projects like this, and they almost always fail not because the developers weren't good or didn't know their stuff, but because there wasn't somebody on the client side who had the political (internal/office-politics, not Democrats/Republicans politics, although within the USG they're often related) capital to get all the little fiefdoms that exist inside a big organization and sit them down and say "Okay, Fuckheads: this is the system we're going to be using, this is how it's going to work, and you will use it."
Projects like this fail when you let every Tom, Dick and Harry start pushing features into it. I've seen situations where software is in the final stages of testing, and somebody decides that it would be fun to bring down the Big Boss to show them where all these millions of dollars have been spent. And the Boss will come down and take one look at the software, and immediately demand that something get changed. Often I don't think that they really care about what they're demanding, they just want to show off that they have the power to change shit, so they do.
It's stuff like that which pushes projects into failure, even if they look dead simple on paper. The problem isn't a software-engineering one, it's a customer-relations one. It's a problem of the people hiring the developers probably not having a good idea of what they wanted in software, and not having a single person in charge of it.
You can tell that happened with this FBI project, because it's obvious just from the summary that the CIO wasn't involved in the project throughout its lifecycle. He just seemingly walked in on it when it was a month away from deployment, at which point I'm sure everything was totally FUBAR. The way to have prevented this would have been to get somebody like that on board from the very beginning, who could have kicked ass and taken names and kept things under control.
Without good leadership on the client side, and a clear set of business processes, requirements, and acceptance criteria, it's not surprising that these large software projects fail as often as they do. However, as long as the failures are equally profitable to the development contracting companies as the successes, they have no problem taking on a contract even though they know the client is going to drive it into the ground and has no idea what they want.
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
Large bureaucratic problems are not solved by contracting out to large, bureaucratic companies. They are solved by small to mid-sized teams working closely with the Agency to assure that 1) everything actually works and 2) the system meets the Agency's needs. Large vendors and consultancies are pressed to deliver quickly and are too scared of losing their next contract to tell the Agency "Hey! This won't have a chance in hell of working!" Smaller teams and companies, while still pressed hard on delivery, generally have the customers' best interests in mind and are willing to tell them when something is wrong.
I can't imagine something like this costing over $100 million dollars. Yes, I realize the importance, complexity, and scale, of the system but, still, $100 million? No way! Just another example of what should be considered criminal activity by a company: bilking the government for all it can just because they can. Such a system could be easily developed for under $5 million dollars and in a year or less by a good, solid team. Cut all the bureaucracy out, focus on the customer (the FBI), and just *work*.
This system could be a vital tool in the war on terror and in crime management in general. Contractors need to take this stuff seriously. It's about more than *just* money. It's about national defense and homeland security (yes, I know, I know). I suppose I'm ranting because I am just tired of seeing this absolute incompetence run rampant through government and I'm tired of watching company after company steal taxpayer dollars by exploiting that incompetence.
I'm sending a letter to Director Mueller and Mr. Amzi today offering a team to do the work for a very fair and affordable price. My team won't be motivated by greed or dollar sign filled eyes but rather by a sense of national pride and a strong desire to really help in the war on crime.
I just wish more companies acted on those things. Revamping all these agencies would be a piece of cake and a LOT more affordable.
-- Anonymous
Although this is true, it is also true that most programmers deliver crap looking, uncomfortable to use, half-assed interfaces if left to their own devices. I know I do...
The same is true of a lot of Open Source software. Fun to do, very powerful, but much of it does look unprofessional or at least unreasonably hard to use, except for other programmers who share the same mindset as the maker.
The tragedy is on one hand that the people who complain about the interface issues are themselves also totally untrained and unqualified to say what exactly needs to be changed, and on the other hand that of course a solid, great looking interface design should be made up front, in the design phase, by professionals. I don't think that ever happens.
But we programmers can start by looking a bit more critically at our own work. A bit. While bitching about those irritating users who think looking professional matters more than actual function. Right?
I believe posters are recognized by their sig. So I made one.
Actually, she does know "shit about fuck": She's smart enough to know she doesn't know. And even better, she's smart enough to to leave you to it! What are you complaining about? I have a boss who doesn't know shit about fuck, but isn't smart enough to know it.
They just "matched" it. That's the industry term. As a stringer for many years (a "stringer" is a type of freelance journalist) I was called by editors many, many times to "match" stories.
You've worked in journalism for, what, a week now? Welcome to the industry. You may want to check with some people in your organisation who've been around the block a few times before firing off embarrassing (to you) letters to the Post Ombudsman.
You help me understand why the mainstream press is in such bad shape these days. Shoddy Lazy Journalism is accepted as standard industry practice.
Religion is the main cause of atheism.
The failure of a effort with the scope of an enterprise-wide project like this in an organization that is clearly "immature" from a project management, technology-management, and IT systems, processes and development standpoint should surprise no one. Never mind achieving enterprise application integration.
Every Fortune 1000 company has hurtled down this obstacle course and has the scars to prove it. It is an expensive journey, but there doesn't seem to be any way around it. As some say, "Pay now or pay later." Multiple iterations of large projects are the norm, not the exception and I believe are an inevitable learning curve that can't be avoided - a rite of passage so to speak. The only disconnect in the FBI's project was thinking that it could be done right the first time at any cost or in any timeframe.
Why? Because half the issues are cultural and derive from the dynamics of the organization which needed to change after 9/11. No vendor or project manager or JAD team can solve for that - they were caught in the middle of a paradigm and culture shift that they had no control over and may not even have been fully aware of or able to articulate and document. The fact that their JAD sessions lasted 6 months is surely proof enough that the organization wasn't ready to talk to IT developers.
Setting aside the lapses in personal competence (e.g. great effort to collect trinkets and souvenirs for scrapbooks while putting a value judgement on valid criticism and calling it "disruptive"), at the end of the day, you can't design what you can't conceive. They didn't know what they didn't know. Now, somewhat older and wiser, perhaps they know quite a bit more, but I doubt they (the FBI organization) know enough to get it right.
One of the "graybeards" in the IEEE article predicts it will be 2010 or 2011. But that will only be the second time. I say it will be more like 2015 for a third try before they will really know with any confidence what they actually knew about the hijackers before 9/11 and have enterprise application systems that are world class and interoperable both within the FBI and externally with other intelligence and law enforcement agencies.