Slashdot Mirror
The FBI Software Upgrade That Wasn't
Davemania writes "Washington Post reports that the FBI's attempt to modernize their department has once again failed. The 170 million dollar Virtual case File system, the agency's second attempt to go paperless is reported to be useless. The finger seems to be pointing at the FBI leadership, greedy contractors and bad software management." From the article: "It appeared to work beautifully. Until Azmi, now the FBI's technology chief, asked about the error rate. Software problem reports, or SPRs, numbered in the hundreds, Azmi recalled in an interview. The problems were multiplying as engineers continued to run tests. Scores of basic functions had yet to be analyzed. 'A month before delivery, you don't have SPRs,' Azmi said. 'You're making things pretty. . . . You're changing colors.'"
Call me crazy, but it sounds like the FBI didn't know what it wanted and SAIC was too scared and proud to play contractor hardball with its client to get the job done. The FBI is legendary for its fractured leadership, fiefdoms (makes most agencies look like a single organism it's so bad) and crap like that.
I've never understood why the government (whose inefficiency in regards to monetary spending has become almost cliche) doesn't set up a system for these sorts of big projects where the funds for it ARE someone's money.
As you said, there would be much more motivation if it wasn't just taxpayer money, so why couldn't they use a system whereby they have several firms fund and set up different solutions and then the best solution gets a predetermined amount of money from the government?
Since the firms would be initially shelling out their own money on the projects without a guarantee of reimbursement, you had better believe they would be busting their asses to make sure the products did what they needed to do quickly and efficiently.
I'm living in a magical dream world, aren't I?
Yes, I've worked both sides of the fence, and quite frankly, the civil service side wasted less, had fewer penpushers, was more rigourous in vetting suppliers, and brought it's project in nearer budget and deadline (that was nearer, not on!)
init 11 - for when you need that edge.
What TFA describes is the current state of general software development for hire, which has changed very little in the 18 years I have been programming.
It doesn't matter how well planned the project is, or how well educated the customer is, or the proper allocation of project champions on the client side, we all end up getting hit with b.s. look-and-feel complaints that end up taking higher priority than fixing bugs.
If you give the client the option between tweaking a template to a report, and tweaking the queries that feed the damn report so it runs 10% faster, the client will ask you to first make it pretty, then worry about the queries. If you dare ask them why, they will give you a b.s. explanation that it is all about perception. That the pretty page looks more "professional" and it looks like more work and care was put into it.
A word of warning to those of you that are new to for-profit programming: whenever somebody uses the "it looks more professional" gambit, it usually means he has no excuse and is hoping you will drop it. He asked you to do it simply to please himself. HE wants the damn color of the page changed, or that heading two pixels taller, etc.
Every couple of years we get hit with new programming methodology fads, but those don't help us with dealing with difficult customers. When you are pulling millions every year from the same two or three government contracts, the last thing your project manager wants is to piss off any of the primaries for the contracts. Extreme programming won't suddenly make your client listen to you.
Why the hell do you think that programmers are so rabidly enthusiastic about working for free for a specific open source project? These same programmers will drag their feet and hate life in general when working at their salaried jobs. At the free project a hell of a lot of the people involved in running the project will actually have a clue, while at the projects at the salaried job the norm is a lot of the people in charge won't have a clue.
Pedro
----
The Insomniac Coder
I'm an editor at IEEE Spectrum. Spectrum laid out out this story in September '05. (I submitted a link to Slashdot at the time, but the editors in their Infinite Wisdom rejected it). Despite our story being prominently featured in google, wikipedia, winning awards, etc, and using similar sources, and so on, the Washington Post didn't acknowledge any of Spectrum's reporting, which has prompted Spectrum's Editor-in-Chief to complain to the Washington's Post's Ombusdman thusly:
Dear Ms. Howell,
We were startled to see that the article "The FBI Upgrade that Wasn't" by
Eggen and Witte in today's Washington Post is taken directly from an article
we did in September 2005 called "Who Killed the Virtual Case File," by Harry
Goldstein (http://www.spectrum.ieee.org/sep05/1455). His article has won 5
major magazine awards. Neither Harry or Spectrum gets credit or attribution
in the Washington Post piece.
Your writers reinterviewed all our sources, including Matthew Patton, whose
only press interview until your story today was in the Spectrum article.
They filed the same FOIA, etc.
Is this plagiarism? Not exactly. Is it shoddy, lazy journalism? You bet.
Sincerely yours,
Susan Hassler
"Just once, I'd like to meet an alien menace that wasn't immune to bullets." -- The Brigadier, Dr. Who
Seriously, I have no idea all of their needs requirements, but it seems like a big one is cross-connecting one set of data with another. The intricate connections of intelligence data probably defies anyones ability to design a system that could capture it all. But, a Wiki, which automatically creates links can do it for you, on the fly. So, create some Wiki templates for information about people, cases, incidents, whatever, and create Wiki links on the keywords when you fill out the templates (names, dates, code names, case numbers, and so on) and let the Wiki link everything together for you.
With a lot of data already entered, in no time you'll be typing in a routine report and find out that the name you just typed already has a Wiki page, and lo and behold! some agent in Nebraska is looking for that exact person for a child abduction. Case closed. All praise the Wiki.
To put it bluntly, the guy in charge of the NOC was (is?) an incompetent jackass. He'd used the same trouble ticket system at his last job and hated it - not because it was bad, but because the admins at his old company had no idea how to run the thing. Long story short, he had one absolute demand before he'd let it be used in "his" NOC: the consultant had to change the window background color from green to blue, because green reminded him of the last installation.
He was serious.
And he actually scheduled a formal compliance test where he would run through the system to make sure he didn't see green anywhere, and informed the consultant and me that if he did, he was rejecting it forever. I was amazed to find that he actually had management backing on this; it's apparently difficult to find managers with obsolete product knowledge, or something like that. So, the company spend a fair number of kilodollars to make the software blue (to the endless delight of the consultant, who drove a nice Corvette and took me to good expense account dinners - which are the best kind!).
Dewey, what part of this looks like authorities should be involved?
Did anyone else pick up on this. from TFA: David Kay, a former SAIC senior vice president who did not work on the program but closely watched its development.... "SAIC was at fault because of the usual contractor reluctance to tell the customer, 'You're screwed up. You don't know what you're doing. This project is going to fail because you're not managing your side of the equation,' " said Kay, who later became the chief U.S. weapons inspector in Iraq. A couple things here i dont get. 1. work for a company that contracts out to the US government. A company that as screwed the pooch since day 1. Then get a JOB from the US government. 2. what the hell does being a VP of a software company give you ANY ability what so ever as a weapons inspector??
MISSING - Sig file. 2 years old black and white and very funny. If found please email me.
15. A complex system that works is invariably found to have evolved from a simple system that works.
16. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system.
IMHO, John Gall's observations on political systems are incredibly apropos to technical systems.
The Apollo Program...sigh I went back to the 1960's
--meh--
Sure, I think we all feel that way after reading this story. But the error could also lie with the Agency. If they are constantly asking for changes and new additions, what can the programmers do.
It seems to me that the idea of doing software as a project is purely fiction. Everybody knows that software has bugs, everybody knows that new features are needed as the landscape changes, and everybody knows that software can be made better. So why do people insist on this flawed idea of a project?
I've come to realize that properly specifying software in advance is unrealistic. People have a tough time thinking through what they actually need a system to do - nobody really knows what they want until they realize that what you have is not it. Then, they'll gladly whine about what's missing.
So I've come to embrace Agile software development as my strategy.
At my small, ASP software company, we don't sell software, we sell its utility. We manage information for school districts, and take all the work out. We do backups, upgrades, maintenance, etc. so the school district can get back to what they do best - teach.
We do upgrades very rapid-fire - often releasing more than once per week. We have a big, huge list of stuff we'd like to do, and as we move forward, we develop whatever's the next most important thing on the list. The list comes primarily from customer whines. We charge hourly rates for development, and basically refuse to bid by the job.
This lets us be VERY flexible as we learn more about the actual needs of the districts we work with - often changing specifications as development is happening. We don't focus on making things "bug free". We focus on fixing bugs rapidly, particularly when they cause a problem for the end user. This lets us get to what's actually needed by the customer FAST. And they LOVE IT!!!
An interesting side-effect of this methodology is that "feature creep" basically disappears - unnecessary features get pushed off because, even if they're cool, they're not what's "needed next" and so get filtered out.
When a change is needed, there's a simple evaluation of "is this important enough to do next?", and this evaluation filters out the crap ideas. Thus, problems like feature creep, bloat, and design by committee, effectively disappear as problems.
I have no problem with your religion until you decide it's reason to deprive others of the truth.
Let's face it, there are lots of problems here:
So what's the solution? Fire the Management responsible for this project.
Management is paid specifically for successful delivery of the project. If they do not deliver the project, they have failed. If they have not been fired, what incentive do they have to make this project succeed? If they cannot be fired, then we've found the fundamental flaw.
You reap what you sow here. If you the taxpayers want union-protected workers who are nearly impossible to fire, then you will get workers who will not be accountable. As long as the rules in government are focused on survival then government workers will continue to CYA (cover your ass), defer decisions and blow money to protect their current position/empire.
And don't blame the government employees here, they're playing the game. The public set the rules and employees play by the rules. People are fired for not playing by the rules. If you feel that the wrong people are (not) being fired, then request that rules of the game be changed. You reap what you sow.
One company I worked for promoted the receptionist to be the project manager for my project (I was the lead developer - and no, they didn't promote me to dev lead from janitor.)
The bad thing about this girl was that she didn't know shit about fuck. The only good thing was that she did realize how ignorant she was, so she didn't question anything I did - just tried to report it to her bosses. And she was good for fetching coffee and ordering stuff (and nice to look at).
Avoid Missing Ball for High Score
her: but X is on the schedule to be compete today. Y and Z are not scheduled to be done until next month. now we are behind schedule.
And what you didn't know, and didn't bother to ask about, was that X had to be done this month because, Q, R and S were all depending on X, and they're major subprojects that are pushing the end date. So the fact that X will be delayed for a couple of weeks will push the entire project end date out, even if it gets Y and Z done a little sooner than planned
Or maybe not. It's also possible that there was no dependency, but if you don't ask, you can't know.
The key is not to make decisions like "I'll do Y and Z now because they'll make X easier, even though X is on the schedule to be done first" without talking to whoever decided that X must be done before Y and Z. If there aren't any dependencies driving X to be done first, that conversation just makes you look really smart, and lets you do the work in the order you wanted to. If there are, however, that conversation keeps you out of hot water.
In general, it's a good idea to get approval before you change task priorities.
Note that there certainly are circumstances in which that doesn't work. I once worked for a PM who was afraid of his own project planning tool, so he basically refused to ever change the plan no matter how much sense it made (even when major requirements changed!). In that case, your best bet is still to do everything just as though your PM weren't a complete fool, and take notes on all of the obviously stupid decisions he makes so that when it all goes down in flames you can make sure he takes the heat, not you. If you have a good line to senior management, you might be able to take your concerns to them early enough that the project doesn't self-destruct, but be very, very careful with that approach. It can screw you in a heartbeat, particularly with a lousy PM who is also a very good political operative.
BTW, in the case of the PM I mentioned, he ultimately got canned. He was going to do something supremely stupid, something that would have serious external repercussions. I took it to his boss, who ordered him not to do that stupid thing. He did it anyway and then tried to cover himself by blaming me, which just compounded the error (duh!).
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
How long have you worked in journalism, Susan?
If someone else does a story, especially a big story like yours, a magazine/newspaper has two options:
1. Reprint your story. Credit you. Pay your organisation money. Look, to their readers, like schmucks because they missed a big story.
Or, and here's what usually happens:
2. Match the story. Re-interview the same sources. Go over the same ground. And then publish a very similar story. This way you not only VERIFY that the original story is true and well reported, but you appear to your readers as if you're out there getting the news.
Shoddy lazy journalism? No. That would have been uncritically reprinting your original story.
They just "matched" it. That's the industry term. As a stringer for many years (a "stringer" is a type of freelance journalist) I was called by editors many, many times to "match" stories.
You've worked in journalism for, what, a week now? Welcome to the industry. You may want to check with some people in your organisation who've been around the block a few times before firing off embarrassing (to you) letters to the Post Ombudsman.
Not sure what context you're talking about, but I'm talking about things from MY context, as the owner of software development and services companies. Sure, I won't be doing any $170 million dollar contracts any time soon, but I have been involved in many $100 million dollar ones.
Also, I'm not responding directly to the specific case of the story, just the parent's generalized statement regarding software projects.
When you get to be involved in a large Government project, there's not much you can do, as it takes on a life of it's own (usually, I imagine a giant ball of snakes in a very deep pit).
Wouldn't that automatically make him overbudget in any publicly traded company or government, where cost is a concern?
I think you're just taking a short/small/unrealistic view at what "cost" is. Sure, my PM might be out of the normal scale for PM's, but I can easily show business cases that justify his "extra" cost (usually less than 2% of the total project, on projects $.5 - $5 million) vs. being late, over budget, and not having something that works. And, I've never had to sell to the "bean counters". I've sold to the client's management, or champions of the project. Sure, bean counters will go over the estimates and projections and costs, and bitch about a whole bunch of stuff, but then we explain and justify those items, and it's all good. I have YET to deal with any client that didn't get it. Let's face it, if I can't sell a higher-than-normal PM to the project, then I'm not doing my job properly. I haven't had a problem doing so in the contracts we've done. And those contracts are with US/CDN Governments and banks. And we deal in very large, complex systems, predominately in globally distributed Oracle installations.
In government and publically traded companies, you don't get to pick your clients. You work with who they tell you to work with. So I guess you've found the way around it: Pick your consituents/customers carefully enough, and you won't have any significantly complex projects to worry about.
Once again, not sure what context you're talking in... I'll assume that it's the Government as client.
Don't kid yourself, governments and public companies FOR SURE have a say in who they work with. When you get right down to it, there is usually ONE GUY/GIRL that is responsible for the project, and has a huge amount to say in who gets the contract. And they generally have the skills (political) to get their way with any kind of oversight that may be in place. To think otherwise would be very naieve, IMO. I have seen sooo many examples of RFP's/contracts/requirements being worded in such a specific manner that only one particular supplier would meet the requirements, just so that the customer could be guaranteed that that supplier would be picked. (I've been on the receiving end of a few of them). I've also seen a large project broken down into many, many smaller ones so that each piece is within the "arbitrary assignment" limit, allowing the manager to authorize the contract himself, without having to go to tender or oversight.
As to the case where I'm the supplier to the client, I guess I'm just used to being at the top of the org chart of a smaller company, because I do exactly that... I pick my clients very, very carefully. As it is, I've declined probably 40% of our potential contracts over the past 2 years because of the potential risk to my company. Unless you're a monster company, like Anderson or IBM, you have to take a serious look at any project you're going to be involved in and evaluate the risk and return on investment. Even then, pick wrong, and it can kill you off. (Enron/Anderson?) And not just the financial ROI... your reputation can make or break you. Go see how well Pangaea is doing around the BC Government these days... their reputation is killing them, slowly but surely.
And it's not about picking the "easy" contracts... we enjoy pushing ourselves as much as anyone who is good at their job. I just wan
$0.02 (CDN)
I tend to agree with this. The similarities to a project I was once involved with are astounding. Millions of dollars were spent over nearly a decade building a system that didn't work. When it became clear that the large companies that had been hired to do the work couldn't get it to work, the people responsible actually did look to a smaller company and highly skilled group of programmers to fix it. In less than a year, for less than a million dollars, a team of 3 people was able to completely rewrite the application. It functioned as it was meant to, was much more efficient than the original design, and IT WORKED. There were a few small bugs that were fixed over the next few years, but they were not critical, workflow impeding issues.