Slashdot Mirror
Social Security Administration Joins Other Agencies With $300M "IT Boondoggle"
alphadogg (971356) writes with news that the SSA has joined the long list of federal agencies with giant failed IT projects. From the article: "Six years ago the Social Security Administration embarked on an aggressive plan to replace outdated computer systems overwhelmed by a growing flood of disability claims. Nearly $300 million later, the new system is nowhere near ready and agency officials are struggling to salvage a project racked by delays and mismanagement, according to an internal report commissioned by the agency. In 2008, Social Security said the project was about two to three years from completion. Five years later, it was still two to three years from being done, according to the report by McKinsey and Co., a management consulting firm. Today, with the project still in the testing phase, the agency can't say when it will be completed or how much it will cost.
Sounds a lot like Beta Slashdot. How many years until that is out of testing and complete?
Legacy Systems are built with 40 years of code and modifications to meet every requirement the user needs.
Then you have 5 years to build something new and try to catch 40 years worth of rules and logic.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
These government agencies need to hire some developers for whom a few million hits is just another day. Something like girlsgonewild.com gets more traffic than healthcare.gov, and handles it with two well-configured commodity servers.
Until the vendors who are building this system get their company name in the headlines, the status quo will continue.
I've never heard of such a thing. Thank goodness Slashdot is here to challenge our preconceptions.
And, now they'll say it was all the fault of the contractor.
In reality, I suspect it's government infighting, poorly defined (and constantly changing) specs, and congress-critters trying to get a piece of the pie for their own districts.
They always blame the contractor but usually it's being managed by incompetent people without enough accountability and controls.
In fairness, I've seen a lot of legacy migrations fail, because it's often damned near impossible to understand the existing system well enough to write a replacement for it, and then you end up breaking everything which has been integrated with it for years.
I've been on a few large legacy replacement projects which fell squarely on their nose as the project progressed, largely because the system is vastly more complex than the initial analysis, and people make it impossible at every turn.
Lost at C:>. Found at C.
stop hiring the free market, it's just grifters.
It would be interesting to know what % of this work was outsourced, or in-sourced, to foreign corporations/workers. Also, it would be interesting to know 1) how contracts for this work were let, and how they were monitored along the way; 2) what incentives for good work were included, or disincentives for bad work were included. Does anyone know?
How man of us have either seen commercials or heard about lawyers colluding with doctors to get people to claim "disability" with the SSA even when they have nothing wrong with them?
This is definitely one of those programs which needs heavy monitoring to weed out waste and fraud, along with military procurement.
True story along the same lines. My dad had to appear in court regarding a disability (non SSA) claim one of the company employees claimed they had and why they couldn't come back to work.
During testimony, a video was shown of this person, who claimed they injured their back, lifting bags of cement over his shoulder and climbing up a ladder to do work. Obviously his claims were rejected and he was fired, but I'm sure we can find thousands of people on SSA "disability" who are doing the same thing.
We will bankrupt ourselves in the vain search for absolute security. -- Dwight D. Eisenhower
Gee, another $300 million down the drain on a system that doesn't work? What a shocker.
Contractors are being well-paid, government supervisors are being well-paid, I'm sure no one will be fired and I'm sure at least some folks who have contributed to the problem are getting bonuses. Just like the banks in 2008 -- there is not a shred of real accountability.
A public that allows this is getting what it pays for. It really has no reason to complain.
So tell us, which are the dishonest and incompetent companies that did the work? Just blacklist them from everywhere.
I find it hard to believe that Lockheed-Martin has had trouble delivering. After all, the F-35 was on time and under budget!
Have none of these places heard of replacing a system piece-by-piece? Or agile development? You don't take a decades old system and replace it in one step. You replace it piece-by-piece. That's not trivial to do, but these stories about "5-year project cancelled with absolutely nothing to show for it" are crazy.
Contractors will give you two basic choices for contracts.
1. Pay for my time. Do what every you want, change what ever you want... but you pay for my time.
2. Specify the project in exact detail and the whole thing will get an over all bid to those specifications. Changes cost extra and may require an additional contract.
I'm assuming the government keeps going with option 1 and I'm thinking most of these issues would go away if they went with option 2.
I've decided to stop wasting my time responding to AC trolls/sockpuppets... so if you want a response from me... login.
And, then contrast that to how much controls were on the people who oversaw it, how well they communicated/knew the requirements, how often they changed them, and how much political infighting they did.
I've been on several projects trying to replace legacy systems. And, as often as not, the client is fighting among themselves, the definitions are either never nailed down or are constantly shifting, and the people involved have no actual experience in managing large scale IT projects.
I'm more likely to think this is a management issue than an issue with who was doing the work.
Ask anybody who has been involved in such a project.
I was on one project that had 11 PMs (no, I'm not kidding), all with their own agenda, and no two of them could ever agree on anything.
It was a truly terrible experience. The people in charge of the existing technology didn't want change and actively sabotaged stuff. The various stakeholders were all trying to carve out their own little fiefdom, the users weren't consulted until late into the project, and the specs might as well have been written in smoke.
The people trying to actually build it were constantly being told "no, don't do that, do this" only to have someone else say "why the hell are you doing this when we told you to do that?". Heck, I've left a meeting one day where everybody said "OK, we agree to do this", only to have a directive come down a day later which said "we can't possibly do that".
Combine that with vastly complex legacy systems nobody really fully understands, because it's been hacked, extended, patched, and a zillion other things for a few decades and you end up with a complete mess.
As I've said elsewhere in this thread, my money is on a failure of the owners of the project to actually take ownership and responsibility, instead of endlessly changing their mind and finding other people to blame. Documenting all of the bullshit becomes a full time job, because you need to CYA for when things go wrong later.
Some problems simply can't be fixed with good technical staff. Because the technical staff is just there to be yelled at and be scapegoats for management incompetence.
Lost at C:>. Found at C.
Also FAIRLY sure displaying videos is a little easier than integrating with a complex database back end which i assume is itself integrated with thousands of health insurance providers who dont have their shit together.
At least it's considerably cheaper than the stupid PPACA exchanges.
The reality is that governments - be it in defence or eslewhere - are always moving the goal posts, and the contractors are running to catch up. So in theory option 2 is the best, but it usually doesn't work out as well as it really ought to. The UK is currently playing the same game with a new system for welfare benefits, and it's equally disasterous. And remember - the private sector is often as bad, they usually get to bury their mistakes without publicity!
Oh good grief. Conservatives have been pushing to simplify things like the tax code that all but virtually eliminate departments like the IRS. Do you ever think these departments are just too huge? Reduce and simplify. It's horrendous and horrible.
The problem is that Specify the project in exact detail is impractical. By the time the details are flushed out, requirements change. And more often than not, the requirements will change while the project is being implemented.
Second, with layers of contracting work, incompetent people are put to work. The screening process is utter bullshit. Fake references. Fake experience. Fake resumes. On-site test would get rid of 95% of crap. But it costs money up front. And it's in the interest of the subcontractor to delay since they are paid by time. There is no incentive for efficiency.
I'm very curious how the racks caused delays. Were there too many that fell over damaging the servers? Perhaps the under performing vendors are being tortured for their failure?
Well, how about that, it appears that the word wrack (synonyms, ruin, destruction, wreckage, (v) to cause destruction) have been replaced by the word rack (synonyms: shelf, torture device, (v) subject to extreme stress).
Bah. My first attempt to be a grammar Nazi and I have to correct myself.
There should be 1 federal IT agency that can do all / most of the government IT with out all of the consulting / contracting / subcontracting overhead.
Unlike Texas, where the state government employs thousands of programmers because they are so liberal. I just got out of a meeting with a bunch of government programmers from Texas. They'll all tell you the same thing - getting stuff done within red tape of a government agency takes them twice as long as long as it took them in the private sector jobs - unless there is a federal grant or contract involved, in which case it takes twenty times as long.
One project they did last year was for a federal government contract, for OSHA. They spent a year and a half developing the system, then during the beta test OSHA cancelled the project. This is after the feds had them write a system where it would print all the database records on paper, to be sent to the feds, who would manually enter it into a computer file, then send that file back to Texas, right back to the same agency who had sent it to them in the first place. That's about typical for the federal government. Government is one thing - it's supposed to be fair and deliberate, not far and efficient. The FEDERAL government is something else entirely.
replacing a large legacy system will take 2-3 years is ignorant or a liar. Neither of which should be involved.
Once again, corporation signed and agreement,. wiggle around for more money and tried to put an organization on the 'it's too late to stop now treadmill'
The Kruger Dunning explains most post on
In a previous life, my company dealt with the SSA data processing side. Obviously I'm posting anonymously. Just wanted to let you know that this is not news.
The SSA data processing and software side is big. So big that you can't do anything without waiting for months to get things through their process. To do anything substantial, you have to get multiple departments together and have their managers agree on the change. They have huge spheres like J2EE, databases, back-end big iron, and so on. Their applications are gigantic. Change management is critical for them, so critical that small changes take a long time. And that's when they're motivated. They have career procrastinators who can turn a 15-minute phone call into a week of back and forth.
What they at SSA do is so important to get right that the change management and testing is everything, and the actual development is almost nothing. It has to be this way. They absolutely have to do it like this. They're not writing $0.99 apps, they're doing something that affects everyone in the USA. (Except the few people who can opt out of SS.) You can write code, but they have to be sure that it works. Their change management process is the single most critical workflow at SSA. Your stuff must pass this process before they'll put it in production, and that means a lot of testing and verification.
BOTTOM LINE: Ain't no way that no contractor is going to whip up no replacement system in no five years, no sir, it ain't gonna happen.
I don't care who it is or how much money they throw at it, the SSA is just too big to do this kind of project successfully in a compressed time frame. I could have told them that 5 years ago and saved people a few million. Besides, $300m isn't that much money for this kind of project.
I can't blame LM for this boondoggle, although I'm sure that they didn't help much. Any big contractor lowballs the bid, takes the money, pays themselves what they want, and hires people with what's left - so I'm sure they scraped the bottom of the barrel for talent on this project.
Hey, the Orioles are having a decent year, so not everything is bad in Baltimore for a change.
If there are any other government agencies out there looking for a new IT system, I can fail to deliver for $100 million.
-
- - You can't take something off the Internet! That's like trying to take pee out of a swimming pool.
The funniest part of your rants is the unfounded assumption that Lockheed is or ever was competent.
Yeah, Conservatives would reduce and simplify government to the point of non-existence, leading us to a system where nobody paid any taxes.
Wouldn't that be grand!
Since we do not have any info on the backend, I suspect this might be an Oracle involved problem. It would be telling if this is so, as I bet Oracle now has very specific clauses that make bad mouthing them a huge legal problem.
If this has nothing to do with Oracle and it's consultants, then it looks like the rest of the consulting industries have take a page from the Oracle handbook on how to get paid and not deliver anything at original estimate.
If these sorts of projects would actually be cheaper if they paid two companies to do them, the one that completes qualification first gets to keep it during the upkeep phase.
Yes, it would require paying for development twice, but you'd be more likely to get a timely result, assuming the upkeep was lucrative and the companies didn't collude.
TFS mentions that the contractor is trying to replace hundreds of different incompatible, overlapping study systems that the government has built or ordered. Does having hundreds of different systems with overlapping functionality trying to talk to ready other sound like proper engineering practice to you? That's what the government decision makers have come up with.
From my experience, government systems are designed for two primary goals. First, give each fiefdom it's piece and second, compliance. Compliance generally means complying with a crap load of old documents written by bureaucrats and lawyers. Actually functioning properly is a distant third on the priority list. Engineered design? Rarely is that mentioned.
Looking at the solictation for bids it's clear that someone at the SSA decided to architect the system before the contractor was selected. This was back in 2009, two years before the contractor was selected.
I can imagine how the RFP cycle went - try to guess what design they had in mind, put together a proposal to do that whether it was a good architecture or not.
The proposed new Disability Case Processing System will be implemented using Service Oriented Architecture (SOA) principles and methods, and will incorporate services, which will be available for use Enterprise wide...The new system is also expected to integrate seamlessly with existing software applications in use by the State DDS's and other agency components as well as access historical data where and when necessary. Further, it must also comply with the SSA'S enterprise architecture and database standards and strategy.
Then spend the next three years trying to do what the proposal says.
I have worked for a number of government agencies and large contractors. This story does not surprise me in the least. What WOULD surprise me is to find a story where one of these large scale IT projects actually got completed on time and on budget. Now there's a story.
Probably a rehash for some but here is, in my opinion, the reasons these projects continue to fail:
1) The procurement and bidding process is flawed, particularly at the federal level. Many firms are forbidden to place bids on projects because that lack this or that credential. As a result, it's the same old players time and time again. IBM. Deloitte. Oracle. Honeywell. Lockheed Martin. Both the Democrats and Republicans know that it's broken and both of them have had a chance to fix it but don't.
2) Governments insist on doing "fixed bid" (i.e. flat rate) projects. In my experience, fixed bid projects almost always turn out poorly, Particularly for large scale projects. What ends up happening is that the project (for whatever reason) falls behind. But the deliverable dates stay the same. Eventually the contractor has to either go back and ask for more money, do the work at a loss, or start cutting corners.
3) Government managers, by and large, don't understand the concept of a budget. Often there are no consequences for finishing late or going over budget. They will still have a job tomorrow.
4) This one happens every time...you go into the project as 50/50 partners with a commitment from the client that they will devote X number of hours per week to the project and hold up their end. The problem is that the government managers have other stuff to do and can't devote X number of hours per week. If you're lucky you get X/2 hours per week. So critical decisions have to wait and delays occur.
5) For some government managers, big big projects are something they have no experience with. Last week they were trying to decide what color coffee maker to get for the break room. This week, they are installing SAP.
6) From the article..."It was supposed to replace 54 separate, antiquated computer systems used by state Social Security offices to process disability claims.". That is not a misprint. 54 antiquated systems. That is a huge undertaking. I would be willing to bet that at least a few of them have no documentation whatsoever. And the only person that knows how to run it retired a few years back. Good luck trying to unravel that mess.
7) The magic bullet theory. Time and time again I hear these management bozos (not just government ones either) spout off about how 'Software package X' is going to revolutionize how you do business..massive efficiency...streamlined processes. Bullshit. The software is only going to be as good as the decisions that are made along the way. And if you make mistakes on a few major decisions these really large software suites (SAP, etc.) can be nearly impossible to change once you start using them.
8) Team members that don't have a stake in the success of the project. Joe in accounting has been using the old antiquated system for the past 23 years. He's retiring in a few years. Do you really think that Joe wants to learn a brand new system? Not fucking likely. He won't get a raise or a bonus for all the extra work and he'll end up having to train his replacement. Meanwhile, his government manager has limited options for those that don't want to play.
9) Managers are unwilling, or unable, to change their convoluted business processes that are the root cause of these un-maintainable systems in the first place. Politics.
10) Lack of documentation. More than once I have been on projects where there is not a single page of documentation describing how the current systems works or what to do if something goes wrong. So you have to sit down and figure it all out yourself and that can take a lot of time.
Typical client exchange:
Me: "So tell me, why do you process Voucher payments in this fashion?"
Client: "Cause that's how Joe showed me to do it when he trained me."
Me: "And why di
Oh good grief. Conservatives have been pushing to simplify things like the tax code that all but virtually eliminate departments like the IRS. Do you ever think these departments are just too huge? Reduce and simplify. It's horrendous and horrible.
Lol. In which country? I assure you it's not the USA.
If you had written "Conservatives have been pretending to push to simplify things like the tax code that all but virtually eliminate departments like the IRS.", then you would get get some nods.
Or if you had said "A Conservative or two had been pushing to simplify things like the tax code that all but virtually eliminate departments like the IRS."
Or are we confusing Conservatives with Ann Rand groupies? They're not the same.
I make no such assumption, that's all you.
I'm saying I'm not willing to conclude the issue was entirely the contractors, and that the people in charge of this quite likely brought their own level of incompetence to the table.
I'm not willing to assume it was entirely the contractor, because I've seen FAR too many examples of management incompetence on these kinds of things.
Lost at C:>. Found at C.
Dangit- I've been in IT for over 30 years, and have never been able to land one of these sweet scams, er, deals...
60 days and Google or Facebook engineers could have it done. Properly. Just saying..