Slashdot Mirror
The FBI Software Upgrade That Wasn't
Davemania writes "Washington Post reports that the FBI's attempt to modernize their department has once again failed. The 170 million dollar Virtual case File system, the agency's second attempt to go paperless is reported to be useless. The finger seems to be pointing at the FBI leadership, greedy contractors and bad software management." From the article: "It appeared to work beautifully. Until Azmi, now the FBI's technology chief, asked about the error rate. Software problem reports, or SPRs, numbered in the hundreds, Azmi recalled in an interview. The problems were multiplying as engineers continued to run tests. Scores of basic functions had yet to be analyzed. 'A month before delivery, you don't have SPRs,' Azmi said. 'You're making things pretty. . . . You're changing colors.'"
I love helping you /.ers out. Instead of spending painstaking hours clicking thru multiple page news stories, I sit here and quickly provide you with printer friendly links
If the anybody can screw up a big project like that it is the government. If it was 170 million of somebody's own money I think that it would have been done a lot better but since it is only the taxpayers money they seem to really mess things up. Perhaps this is one of the many reason we should limit the federal govt to their proper role as given in the Constitution.
Personally, I'd prefer the FBI not go paperless. Because (a) paper trails are nice in investigations and such (y'know, when the FBI finally goes up against the Supreme Court) and (b) stuff that doesn't have a hardcopy tends to get lost more often than physical objects...especially embarassing things...especially by government agencies.
Yes. I'm slightly paranoid.
Anyone else think the comments just weren't rendering right before they turned off ABP and saw ads?
And that is how you get rich doing work for the government. The government agency comes up with a half-assed plan, you put in a low bid, they accept and start handing you checks, and you make things look pretty, all the while hiding the flaws. In then end, you've become rich, the goverment runs a deficit, and the American taxpayer foots the bill.
GetOuttaMySpace - The Anti-Social Network
'A month before delivery,' Professor Knuth said looking up through his spectacles 'you can start implementing it if your correctness proofs are complete.'"
Ha! Welcome to the real world, guys.
Call me crazy, but it sounds like the FBI didn't know what it wanted and SAIC was too scared and proud to play contractor hardball with its client to get the job done. The FBI is legendary for its fractured leadership, fiefdoms (makes most agencies look like a single organism it's so bad) and crap like that.
Slashdot Burying Stories About Slashdot Media Owned
It's called WOM, or Write-Only Memory system. This system has near-infinite storage capacity, and can be implemented across the entire enterprise.
Document retrival in the WOM? Not a problem! Just create imaginary documents! Isin't that the way it's done, anyway?
Oh, and if you need a record expunged, not a problem! In fact, it requires almost no effort at all!
Write-Only Memory Virtual Filing System. It was good enough for Nasa, it ought to be good enough for the FBI.
Ruby Neural Evolution of Augmenting Topologies
That quote was just too funny. In any huge huge huge project the state of play a month before is usually going through the thousands of trouble reports, deciding which features you will turn off so you can ge the release out, figuring out work arounds for the rest of them, discovering that the analysis was flawed on some functions and of course by now you ditched many other features because the analysis was not done at all. And yes you change a few icon colors to keep key users happy and get your sign-off. This is for a successful 1.0 implementation.
Wow... I have never, ever seen a software product that wasn't working on QA bug reports right up to the minute the gold disc is burned. And afterwards, of course, working on all the pre-release bugs that had been classified as 'known issues'.
Chelloveck
I give up on debugging. From now on, SIGSEGV is a feature.
Spit.
The trouble with project managers (and security people) is that they have a checklist mentality.
PM: Have you done this as yet.
You: No, there is no need for it
PM: But I need to get it checked off on my plan
You: It shouldn't be on the plan in the first place
PM: But it is on the plan, so I need to get it checked off. When are you going to do it.
And so on.
Rereading the summary, the submitter has it wrong - "FBI's attempt to modernize their department has once again failed" implies that Sentinel has failed - which is definitely not the content of the article. Even the snippet quoted is about VCF having problems, not Sentinel.
It would be nice if, sometimes, large organizations realized that applying computers to solving the problems of a paper trail is going to cause many many problems before any benefits are seen. In working with my university, I've seen time and again the tendency of higher-ups to see computers as a panacea to any/all problems an office might confront in keeping records on things.
For example, our housing lottery system was, until this past year, an in-person process where people were assigned times, showed up, claimed rooms, and was a fair system that worked. Then, the university got all fancy pants and replaced that lottery with this unbelievably crappy system called Residential Management System. To use: kill ad blocker, only use it in IE for Windows, ensure javascript settings are correct, and then wait until the clock allows you into the online lottery system. Attempt to use a non-intuitive UI that is completely new because you couldn't look at it before while time ticks away and other people claim the rooms you wanted. Even though I got the room I wanted, the experience was horrifyingly bad.
For these large organizations, I think less can be more. Keep your paper trail, but create a highly efficient system for digitizing documents. That way, you start to have some advantages of computers (search, organization, cross-referencing) without the liability of a completely paperless system. From here, you can slowly make a transition from leaning on paper to leaning on machines. But that would be the sane way of doing things, and we're talking about a governement organization here.
The new project is even worse than the old. No software, with the possible exception of truly safety-critical stuff like missle-control or nuclear power plants, needs to cost $425 million and take four years. You could have a custom OS written in pure assembly for a quarter of that!
Media that can be recorded and distributed can be recorded and distributed.
-kfg
What TFA describes is the current state of general software development for hire, which has changed very little in the 18 years I have been programming.
It doesn't matter how well planned the project is, or how well educated the customer is, or the proper allocation of project champions on the client side, we all end up getting hit with b.s. look-and-feel complaints that end up taking higher priority than fixing bugs.
If you give the client the option between tweaking a template to a report, and tweaking the queries that feed the damn report so it runs 10% faster, the client will ask you to first make it pretty, then worry about the queries. If you dare ask them why, they will give you a b.s. explanation that it is all about perception. That the pretty page looks more "professional" and it looks like more work and care was put into it.
A word of warning to those of you that are new to for-profit programming: whenever somebody uses the "it looks more professional" gambit, it usually means he has no excuse and is hoping you will drop it. He asked you to do it simply to please himself. HE wants the damn color of the page changed, or that heading two pixels taller, etc.
Every couple of years we get hit with new programming methodology fads, but those don't help us with dealing with difficult customers. When you are pulling millions every year from the same two or three government contracts, the last thing your project manager wants is to piss off any of the primaries for the contracts. Extreme programming won't suddenly make your client listen to you.
Why the hell do you think that programmers are so rabidly enthusiastic about working for free for a specific open source project? These same programmers will drag their feet and hate life in general when working at their salaried jobs. At the free project a hell of a lot of the people involved in running the project will actually have a clue, while at the projects at the salaried job the norm is a lot of the people in charge won't have a clue.
Pedro
----
The Insomniac Coder
I'm an editor at IEEE Spectrum. Spectrum laid out out this story in September '05. (I submitted a link to Slashdot at the time, but the editors in their Infinite Wisdom rejected it). Despite our story being prominently featured in google, wikipedia, winning awards, etc, and using similar sources, and so on, the Washington Post didn't acknowledge any of Spectrum's reporting, which has prompted Spectrum's Editor-in-Chief to complain to the Washington's Post's Ombusdman thusly:
Dear Ms. Howell,
We were startled to see that the article "The FBI Upgrade that Wasn't" by
Eggen and Witte in today's Washington Post is taken directly from an article
we did in September 2005 called "Who Killed the Virtual Case File," by Harry
Goldstein (http://www.spectrum.ieee.org/sep05/1455). His article has won 5
major magazine awards. Neither Harry or Spectrum gets credit or attribution
in the Washington Post piece.
Your writers reinterviewed all our sources, including Matthew Patton, whose
only press interview until your story today was in the Spectrum article.
They filed the same FOIA, etc.
Is this plagiarism? Not exactly. Is it shoddy, lazy journalism? You bet.
Sincerely yours,
Susan Hassler
"Just once, I'd like to meet an alien menace that wasn't immune to bullets." -- The Brigadier, Dr. Who
We've learned this over and over again at my company. The likelihood of scrapping the whole thing because you've got nothing is logarithmic to the cost. That is, the more the costs go up, the more likely you scrap the whole thing.
The project has to be bitten in chunks. Lay out the functionality, and then start implementing it one small piece at a time, integrating as you go along. The Big Bang approach is always doomed to failure, or explosive costs, especially when you get to the reality that to deploy you need to shut down the business for two weeks to manage the data conversion. Lot's of small $1 million projects are more likely to succeed and be at budget then one big $20 million project.
This isn't news. It's the whole momentum behind a lot of modern development techniques such as Agile, or architectural such as SOA.
There's also a corrolary that any project involving a big consulting company like EDS, CSC, Anderson(or whatever the hell their name isnow), etc. is more than likely going to cost double what it should.
To put it bluntly, the guy in charge of the NOC was (is?) an incompetent jackass. He'd used the same trouble ticket system at his last job and hated it - not because it was bad, but because the admins at his old company had no idea how to run the thing. Long story short, he had one absolute demand before he'd let it be used in "his" NOC: the consultant had to change the window background color from green to blue, because green reminded him of the last installation.
He was serious.
And he actually scheduled a formal compliance test where he would run through the system to make sure he didn't see green anywhere, and informed the consultant and me that if he did, he was rejecting it forever. I was amazed to find that he actually had management backing on this; it's apparently difficult to find managers with obsolete product knowledge, or something like that. So, the company spend a fair number of kilodollars to make the software blue (to the endless delight of the consultant, who drove a nice Corvette and took me to good expense account dinners - which are the best kind!).
Dewey, what part of this looks like authorities should be involved?
full disclosure: I wrote the "Who Killed the Virtual Case File" story for Spectrum, which ran last September.
Here's some more food for thought about the "reporting" behind the FBI story:
What's the news angle that warrants front page attention in the Post? That the Post reporters obtained the "unreleased" Aerospace report? Not news: the report was released to Spectrum at the end of April after nine months of litigating a Freedom of Information Act Request.
All the Post reporters had to do was google "virtual case file" and voila! the story pops up as number 1, right there for them to rewrite!
But say they are too lazy to bother googling. They just want the summary. The Spectrum article is the basis for the Wikipedia Entry on the Virtual Case File and the only external link. http://en.wikipedia.org/wiki/Virtual_Case_File
The Spectrum article was the first and until the Post article, the only one to mention Matthew Patton, who was unearthed by dint of investigative reporting nowhere acknowledged in the Post article.
The Post article purports to turn a spotlight on SAIC, in part by quoting David Kay, the Iraq weapons inspector, who was a former SAIC VP--but who had absolutely no firsthand knowledge of the VCF project.
The Post article uncritcally takes FBI CIO Azmi's word that the follow up project Sentinel is on-budget and on-time, when other news outlets have recently reported about a growing sense within the FBI that this project is doomed to a fate similar to the VCF's.
"I love helping you /.ers out."
Can I borrow $50 from you?
This is how big government projects *should* be done. Hire a good contractor, set a minimum and then give bonuses for good performance and penalties for bad. Did the final tally cost a lot in bonuses? Yes. Was it worth it? Yes- they fixed a major problem in amazing time and did it correctly, plus they had a bunch of blue-collar folks make serious coin working triple time, all of which got plowed back into the local economy.
You can argue it wasn't on budget due to the bonuses, but it was assumed from the beginning they'd be paying out. Since the daily economic loss to LA was higher than the daily bonus for finishing early, I'd argue it was actually under budget.
"Seven Deadly Sins? I thought it was a to-do list!"
Heck, give me half that--$85 million--and I'll develop the friggin' system myself.
/rant
You'd probably think so, but I bet after the first few months of totally contradictory change requests, specification creep, and an utter lack of hard-and-fast acceptance criteria, that you'd throw up your hands, too.
You can blame the contractors all you want, but I've worked on a bunch of projects like this, and they almost always fail not because the developers weren't good or didn't know their stuff, but because there wasn't somebody on the client side who had the political (internal/office-politics, not Democrats/Republicans politics, although within the USG they're often related) capital to get all the little fiefdoms that exist inside a big organization and sit them down and say "Okay, Fuckheads: this is the system we're going to be using, this is how it's going to work, and you will use it."
Projects like this fail when you let every Tom, Dick and Harry start pushing features into it. I've seen situations where software is in the final stages of testing, and somebody decides that it would be fun to bring down the Big Boss to show them where all these millions of dollars have been spent. And the Boss will come down and take one look at the software, and immediately demand that something get changed. Often I don't think that they really care about what they're demanding, they just want to show off that they have the power to change shit, so they do.
It's stuff like that which pushes projects into failure, even if they look dead simple on paper. The problem isn't a software-engineering one, it's a customer-relations one. It's a problem of the people hiring the developers probably not having a good idea of what they wanted in software, and not having a single person in charge of it.
You can tell that happened with this FBI project, because it's obvious just from the summary that the CIO wasn't involved in the project throughout its lifecycle. He just seemingly walked in on it when it was a month away from deployment, at which point I'm sure everything was totally FUBAR. The way to have prevented this would have been to get somebody like that on board from the very beginning, who could have kicked ass and taken names and kept things under control.
Without good leadership on the client side, and a clear set of business processes, requirements, and acceptance criteria, it's not surprising that these large software projects fail as often as they do. However, as long as the failures are equally profitable to the development contracting companies as the successes, they have no problem taking on a contract even though they know the client is going to drive it into the ground and has no idea what they want.
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
They just "matched" it. That's the industry term. As a stringer for many years (a "stringer" is a type of freelance journalist) I was called by editors many, many times to "match" stories.
You've worked in journalism for, what, a week now? Welcome to the industry. You may want to check with some people in your organisation who've been around the block a few times before firing off embarrassing (to you) letters to the Post Ombudsman.
You help me understand why the mainstream press is in such bad shape these days. Shoddy Lazy Journalism is accepted as standard industry practice.
Religion is the main cause of atheism.