Slashdot Mirror
The FBI Software Upgrade That Wasn't
Davemania writes "Washington Post reports that the FBI's attempt to modernize their department has once again failed. The 170 million dollar Virtual case File system, the agency's second attempt to go paperless is reported to be useless. The finger seems to be pointing at the FBI leadership, greedy contractors and bad software management." From the article: "It appeared to work beautifully. Until Azmi, now the FBI's technology chief, asked about the error rate. Software problem reports, or SPRs, numbered in the hundreds, Azmi recalled in an interview. The problems were multiplying as engineers continued to run tests. Scores of basic functions had yet to be analyzed. 'A month before delivery, you don't have SPRs,' Azmi said. 'You're making things pretty. . . . You're changing colors.'"
I love helping you /.ers out. Instead of spending painstaking hours clicking thru multiple page news stories, I sit here and quickly provide you with printer friendly links
If the anybody can screw up a big project like that it is the government. If it was 170 million of somebody's own money I think that it would have been done a lot better but since it is only the taxpayers money they seem to really mess things up. Perhaps this is one of the many reason we should limit the federal govt to their proper role as given in the Constitution.
Personally, I'd prefer the FBI not go paperless. Because (a) paper trails are nice in investigations and such (y'know, when the FBI finally goes up against the Supreme Court) and (b) stuff that doesn't have a hardcopy tends to get lost more often than physical objects...especially embarassing things...especially by government agencies.
Yes. I'm slightly paranoid.
Anyone else think the comments just weren't rendering right before they turned off ABP and saw ads?
'A month before delivery, you don't have SPRs,' Azmi said. 'You're making things pretty. . . . You're changing colors.'
Can I get the icon in 'cornflower blue'?
____
~ |rip/\/\aster /\/\onkey
And that is how you get rich doing work for the government. The government agency comes up with a half-assed plan, you put in a low bid, they accept and start handing you checks, and you make things look pretty, all the while hiding the flaws. In then end, you've become rich, the goverment runs a deficit, and the American taxpayer foots the bill.
GetOuttaMySpace - The Anti-Social Network
'A month before delivery,' Professor Knuth said looking up through his spectacles 'you can start implementing it if your correctness proofs are complete.'"
Ha! Welcome to the real world, guys.
"I'll just file this case in my Virtual Filing Cabinet"
"You're everywhere. You're omnivorous."
Call me crazy, but it sounds like the FBI didn't know what it wanted and SAIC was too scared and proud to play contractor hardball with its client to get the job done. The FBI is legendary for its fractured leadership, fiefdoms (makes most agencies look like a single organism it's so bad) and crap like that.
What do you expect? They don't have time or resources for testing because all the agents are too busy listening in on my calls to my grandmother.
Are you...Are you some kind of genius?
No, ma'am, I'm just a regular Slashdot reader.
They should have just started by picking a decent directory structure for the documents and then hooking up a decent search engine like the Google Appliance. Then the users could simply use web browsers instead of a weak, buggy, and expensive custom application.
Non CS people who commission custom software development often have no clue how expensive their ego driven non-standard features can be.
Slashdot Burying Stories About Slashdot Media Owned
It's called WOM, or Write-Only Memory system. This system has near-infinite storage capacity, and can be implemented across the entire enterprise.
Document retrival in the WOM? Not a problem! Just create imaginary documents! Isin't that the way it's done, anyway?
Oh, and if you need a record expunged, not a problem! In fact, it requires almost no effort at all!
Write-Only Memory Virtual Filing System. It was good enough for Nasa, it ought to be good enough for the FBI.
Ruby Neural Evolution of Augmenting Topologies
"You're making things pretty. . . . You're changing colors."
That's the FBI policy: they're part of Homeland Security, so their job is mainly to tells what color today is. Otherwise terrorists might have trouble knowing which days we're not checking everyone or paying closest attention.
--
make install -not war
Wow... I have never, ever seen a software product that wasn't working on QA bug reports right up to the minute the gold disc is burned. And afterwards, of course, working on all the pre-release bugs that had been classified as 'known issues'.
Chelloveck
I give up on debugging. From now on, SIGSEGV is a feature.
Spit.
The trouble with project managers (and security people) is that they have a checklist mentality.
PM: Have you done this as yet.
You: No, there is no need for it
PM: But I need to get it checked off on my plan
You: It shouldn't be on the plan in the first place
PM: But it is on the plan, so I need to get it checked off. When are you going to do it.
And so on.
Rereading the summary, the submitter has it wrong - "FBI's attempt to modernize their department has once again failed" implies that Sentinel has failed - which is definitely not the content of the article. Even the snippet quoted is about VCF having problems, not Sentinel.
It would be nice if, sometimes, large organizations realized that applying computers to solving the problems of a paper trail is going to cause many many problems before any benefits are seen. In working with my university, I've seen time and again the tendency of higher-ups to see computers as a panacea to any/all problems an office might confront in keeping records on things.
For example, our housing lottery system was, until this past year, an in-person process where people were assigned times, showed up, claimed rooms, and was a fair system that worked. Then, the university got all fancy pants and replaced that lottery with this unbelievably crappy system called Residential Management System. To use: kill ad blocker, only use it in IE for Windows, ensure javascript settings are correct, and then wait until the clock allows you into the online lottery system. Attempt to use a non-intuitive UI that is completely new because you couldn't look at it before while time ticks away and other people claim the rooms you wanted. Even though I got the room I wanted, the experience was horrifyingly bad.
For these large organizations, I think less can be more. Keep your paper trail, but create a highly efficient system for digitizing documents. That way, you start to have some advantages of computers (search, organization, cross-referencing) without the liability of a completely paperless system. From here, you can slowly make a transition from leaning on paper to leaning on machines. But that would be the sane way of doing things, and we're talking about a governement organization here.
The new project is even worse than the old. No software, with the possible exception of truly safety-critical stuff like missle-control or nuclear power plants, needs to cost $425 million and take four years. You could have a custom OS written in pure assembly for a quarter of that!
Media that can be recorded and distributed can be recorded and distributed.
-kfg
What TFA describes is the current state of general software development for hire, which has changed very little in the 18 years I have been programming.
It doesn't matter how well planned the project is, or how well educated the customer is, or the proper allocation of project champions on the client side, we all end up getting hit with b.s. look-and-feel complaints that end up taking higher priority than fixing bugs.
If you give the client the option between tweaking a template to a report, and tweaking the queries that feed the damn report so it runs 10% faster, the client will ask you to first make it pretty, then worry about the queries. If you dare ask them why, they will give you a b.s. explanation that it is all about perception. That the pretty page looks more "professional" and it looks like more work and care was put into it.
A word of warning to those of you that are new to for-profit programming: whenever somebody uses the "it looks more professional" gambit, it usually means he has no excuse and is hoping you will drop it. He asked you to do it simply to please himself. HE wants the damn color of the page changed, or that heading two pixels taller, etc.
Every couple of years we get hit with new programming methodology fads, but those don't help us with dealing with difficult customers. When you are pulling millions every year from the same two or three government contracts, the last thing your project manager wants is to piss off any of the primaries for the contracts. Extreme programming won't suddenly make your client listen to you.
Why the hell do you think that programmers are so rabidly enthusiastic about working for free for a specific open source project? These same programmers will drag their feet and hate life in general when working at their salaried jobs. At the free project a hell of a lot of the people involved in running the project will actually have a clue, while at the projects at the salaried job the norm is a lot of the people in charge won't have a clue.
Pedro
----
The Insomniac Coder
I'm an editor at IEEE Spectrum. Spectrum laid out out this story in September '05. (I submitted a link to Slashdot at the time, but the editors in their Infinite Wisdom rejected it). Despite our story being prominently featured in google, wikipedia, winning awards, etc, and using similar sources, and so on, the Washington Post didn't acknowledge any of Spectrum's reporting, which has prompted Spectrum's Editor-in-Chief to complain to the Washington's Post's Ombusdman thusly:
Dear Ms. Howell,
We were startled to see that the article "The FBI Upgrade that Wasn't" by
Eggen and Witte in today's Washington Post is taken directly from an article
we did in September 2005 called "Who Killed the Virtual Case File," by Harry
Goldstein (http://www.spectrum.ieee.org/sep05/1455). His article has won 5
major magazine awards. Neither Harry or Spectrum gets credit or attribution
in the Washington Post piece.
Your writers reinterviewed all our sources, including Matthew Patton, whose
only press interview until your story today was in the Spectrum article.
They filed the same FOIA, etc.
Is this plagiarism? Not exactly. Is it shoddy, lazy journalism? You bet.
Sincerely yours,
Susan Hassler
"Just once, I'd like to meet an alien menace that wasn't immune to bullets." -- The Brigadier, Dr. Who
Seriously, I have no idea all of their needs requirements, but it seems like a big one is cross-connecting one set of data with another. The intricate connections of intelligence data probably defies anyones ability to design a system that could capture it all. But, a Wiki, which automatically creates links can do it for you, on the fly. So, create some Wiki templates for information about people, cases, incidents, whatever, and create Wiki links on the keywords when you fill out the templates (names, dates, code names, case numbers, and so on) and let the Wiki link everything together for you.
With a lot of data already entered, in no time you'll be typing in a routine report and find out that the name you just typed already has a Wiki page, and lo and behold! some agent in Nebraska is looking for that exact person for a child abduction. Case closed. All praise the Wiki.
From the original post: the FBI's attempt to modernize their department has once again failed
Failed once again? The article (you have to read the whole thing) says it's on track.
The article is 90% about the Virtual Case File system ("built" by SAIC) and it's eventual demise in early 2005, almost 2 years ago. At the end, they discuss the FBI's replacement for VCF, saying:
"Last year, FBI officials announced a replacement for VCF, named Sentinel, that is projected to cost $425 million and will not be fully operational until 2009. A temporary overlay version of the software, however, is planned for launch next year. The project's main contractor, Lockheed Martin Corp., will be paid $305 million and will be required to meet benchmarks as the project proceeds. FBI officials say Sentinel has survived three review sessions and is on budget and on schedule."
We've learned this over and over again at my company. The likelihood of scrapping the whole thing because you've got nothing is logarithmic to the cost. That is, the more the costs go up, the more likely you scrap the whole thing.
The project has to be bitten in chunks. Lay out the functionality, and then start implementing it one small piece at a time, integrating as you go along. The Big Bang approach is always doomed to failure, or explosive costs, especially when you get to the reality that to deploy you need to shut down the business for two weeks to manage the data conversion. Lot's of small $1 million projects are more likely to succeed and be at budget then one big $20 million project.
This isn't news. It's the whole momentum behind a lot of modern development techniques such as Agile, or architectural such as SOA.
There's also a corrolary that any project involving a big consulting company like EDS, CSC, Anderson(or whatever the hell their name isnow), etc. is more than likely going to cost double what it should.
If the specifications for the system were imprecise or constantly changing (as often happens), that would limit the ability of ANY software developer to create a stable functional system on time and within budget.
I'm not going to criticise the folks who were trying to implement the system until I know a lot more about the actual conditions in which they were trying to work...
Mainframe/UNIX Bit Twiddler and long time Windows/Linux Hobbyist.
The Theorem Theorem: If If, Then Then.
full disclosure: I wrote the "Who Killed the Virtual Case File" story for Spectrum, which ran last September.
Here's some more food for thought about the "reporting" behind the FBI story:
What's the news angle that warrants front page attention in the Post? That the Post reporters obtained the "unreleased" Aerospace report? Not news: the report was released to Spectrum at the end of April after nine months of litigating a Freedom of Information Act Request.
All the Post reporters had to do was google "virtual case file" and voila! the story pops up as number 1, right there for them to rewrite!
But say they are too lazy to bother googling. They just want the summary. The Spectrum article is the basis for the Wikipedia Entry on the Virtual Case File and the only external link. http://en.wikipedia.org/wiki/Virtual_Case_File
The Spectrum article was the first and until the Post article, the only one to mention Matthew Patton, who was unearthed by dint of investigative reporting nowhere acknowledged in the Post article.
The Post article purports to turn a spotlight on SAIC, in part by quoting David Kay, the Iraq weapons inspector, who was a former SAIC VP--but who had absolutely no firsthand knowledge of the VCF project.
The Post article uncritcally takes FBI CIO Azmi's word that the follow up project Sentinel is on-budget and on-time, when other news outlets have recently reported about a growing sense within the FBI that this project is doomed to a fate similar to the VCF's.
"I love helping you /.ers out."
Can I borrow $50 from you?
This is how big government projects *should* be done. Hire a good contractor, set a minimum and then give bonuses for good performance and penalties for bad. Did the final tally cost a lot in bonuses? Yes. Was it worth it? Yes- they fixed a major problem in amazing time and did it correctly, plus they had a bunch of blue-collar folks make serious coin working triple time, all of which got plowed back into the local economy.
You can argue it wasn't on budget due to the bonuses, but it was assumed from the beginning they'd be paying out. Since the daily economic loss to LA was higher than the daily bonus for finishing early, I'd argue it was actually under budget.
"Seven Deadly Sins? I thought it was a to-do list!"
Heck, give me half that--$85 million--and I'll develop the friggin' system myself.
/rant
You'd probably think so, but I bet after the first few months of totally contradictory change requests, specification creep, and an utter lack of hard-and-fast acceptance criteria, that you'd throw up your hands, too.
You can blame the contractors all you want, but I've worked on a bunch of projects like this, and they almost always fail not because the developers weren't good or didn't know their stuff, but because there wasn't somebody on the client side who had the political (internal/office-politics, not Democrats/Republicans politics, although within the USG they're often related) capital to get all the little fiefdoms that exist inside a big organization and sit them down and say "Okay, Fuckheads: this is the system we're going to be using, this is how it's going to work, and you will use it."
Projects like this fail when you let every Tom, Dick and Harry start pushing features into it. I've seen situations where software is in the final stages of testing, and somebody decides that it would be fun to bring down the Big Boss to show them where all these millions of dollars have been spent. And the Boss will come down and take one look at the software, and immediately demand that something get changed. Often I don't think that they really care about what they're demanding, they just want to show off that they have the power to change shit, so they do.
It's stuff like that which pushes projects into failure, even if they look dead simple on paper. The problem isn't a software-engineering one, it's a customer-relations one. It's a problem of the people hiring the developers probably not having a good idea of what they wanted in software, and not having a single person in charge of it.
You can tell that happened with this FBI project, because it's obvious just from the summary that the CIO wasn't involved in the project throughout its lifecycle. He just seemingly walked in on it when it was a month away from deployment, at which point I'm sure everything was totally FUBAR. The way to have prevented this would have been to get somebody like that on board from the very beginning, who could have kicked ass and taken names and kept things under control.
Without good leadership on the client side, and a clear set of business processes, requirements, and acceptance criteria, it's not surprising that these large software projects fail as often as they do. However, as long as the failures are equally profitable to the development contracting companies as the successes, they have no problem taking on a contract even though they know the client is going to drive it into the ground and has no idea what they want.
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
Actually, Susan is my boss, but I'm going to assume you paid attention to where I indicated that in my original post, and are addressing your comments rhetorically. For the record Susan's been in journalism for decades, is a frequent judge for journalism awards, lectured at NYU's journalism school, etc, etc., and been the EIC of Spectrum for over six years. So drop the patronizing smarm. Finding your own angle on a story that's going around is one thing: failing to give adequate attribution is another, and is violation of, e.g., the Washington Post's ethics policy:
... It is the policy of this newspaper to give credit to other publications that develop exclusive stories worthy of coverage by The Post.
Attribution of material from other newspapers and other media must be total.
Certainly, for example, digging up Matthew Patton was an element of the VCF story that was exclusive to Spectrum's coverage, as Patton had not appeared in other media outlets before or since Spectrum's coverage until today.
Even when publications are chasing the same story, when one publication gets something unique it is normal to see lines such as "As first reported in the New York Times..." etc in stories in other outlets. A similar attribution in passing in the text was all that would have been required: instead the only attempt at attribution by the Post article is buried in the credits list for the accompanying timeline graphic, where the "Institute of Electrical and Electronic Engineers" is credited as a source, which is a) insufficient and b) wrong (the source was "IEEE Spectrum Magazine". Crediting the IEEE is like crediting General Electric for information taken from a "Today Show" segment.)
As a concrete example, let's look at the recent Sony-BMG DRM rootkit controversy. I did a story on that, interviewing many of the people involved, people who got interviewed by a lot of media outlets at the same time, but when I found a nugget that had been exclusivey reported by one other news outlet--a video of a DHS offcial talking to a local buisiness group about the issue--I gave credit where credit where was due. To the Washington Post in fact: "One party that cares is the U.S. Department of Homeland Security, which includes cybersecurity as part of its portfolio. On 10 November, as reported by the Washington Post, Stewart Baker, assistant secretary for homeland security, made a pointed reference to the Sony BMG protection system..." [Emphasis added]
Speaking personally as someone who hires freelancers, and who's been a staff journalist and editor for somewhat more than a week myself, if your post is indicative of your grasp of the ethical standards of journalism, you can be sure this is one editor who wouldn't call on your abilities as a stringer, or anything else.
"Just once, I'd like to meet an alien menace that wasn't immune to bullets." -- The Brigadier, Dr. Who
They just "matched" it. That's the industry term. As a stringer for many years (a "stringer" is a type of freelance journalist) I was called by editors many, many times to "match" stories.
You've worked in journalism for, what, a week now? Welcome to the industry. You may want to check with some people in your organisation who've been around the block a few times before firing off embarrassing (to you) letters to the Post Ombudsman.
You help me understand why the mainstream press is in such bad shape these days. Shoddy Lazy Journalism is accepted as standard industry practice.
Religion is the main cause of atheism.