The Billion-Dollar Website

stoborrobots writes: The Government Accountability Office has investigated the cost blowouts associated with how the Centers for Medicare & Medicaid Services (CMS) handled the Healthcare.gov project. It has released a 60-page report entitled Healthcare.gov: Ineffective Planning and Oversight Practices Underscore the Need for Improved Contract Management, with a 5 page summary. The key takeaway messages are:

• CMS undertook the development of Healthcare.gov and its related systems without effective planning or oversight practices...
• [The task] was a complex effort with compressed time frames. To be expedient, CMS issued task orders ... when key technical requirements were unknown...
• CMS identified major performance issues ... but took only limited steps to hold the contractor accountable.
• CMS awarded a new contract to another firm [and the new contract's cost has doubled] due to changes such as new requirements and other enhancements...

Comments

Technical People

[TechyImmigrant]

Non technical people are not competent to commission technical work from technical people.

If you (as a government or large company) don't have your own technical people on staff to oversee the process and comprehend or write the specs, you're doomed. The contractors know well how to milk a cash cow, simply by adhering to the specs written by people who don't understand how to write specs.

Re:Technical People

[Chrisq]

Non technical people are not competent to commission technical work from technical people.

If you (as a government or large company) don't have your own technical people on staff to oversee the process and comprehend or write the specs, you're doomed. The contractors know well how to milk a cash cow, simply by adhering to the specs written by people who don't understand how to write specs.

Sadly this is true, but it shouldn't be. Technical people should have the professionalism to analyse requirements and check that the requirements fit the purpose. Unfortunately the way of the world is that technical people would be quickly shuffled out of the way by sales and marketing if they started to reduce revenue by telling a customer what they really wanted instead of what the spec says.

Re:Technical People

[Anonymous Coward]

Technical people should have the professionalism to analyse requirements and check that the requirements fit the purpose.

Typically, they do. However you overlook one key component of this and then dump the blame completely on sales & marketing (not entirely unfair, they are typically huge scumbags). This requirements analysis and design phase costs more money than development. The cost for architecting software is far higher than simply building it. Clients typically do not want to pay for this and assume they know how to do it themselves. This is exactly what happened to healthcare.gov.

I have seen this happen with both state government and private corporation projects alike. I've never done a federal project, so I can't speak first hand about that, but I know people who have and they report the same is true when working for a federal agency.

So yes, part of the blame definitely should go to the sales & marketing bastards, but a very large chunk is on the client for not wanting to fork over the cash up front. This almost always results in spending even more cash later on to fix what people think are bugs but are really design failures which result from poor architecture and design processes.

Re:Technical People

[david.emery]

PLEASE Mod Parent up! I've been working on large government funded systems (defense and commercial) for 35+ years, and in my view programs are screwed from the beginning by overly-aggressive schedules for the up-front work. When the incomplete/absent requirements/architecture/design results in coding, or more often test and integration delays, they'll find more money and time. By then, it's too late.

Back when we had explicit waterfall milestones (requirements review, preliminary design review, etc), we could tell at PDR a program would fail as a result of incomplete or even incorrect requirements & architecture.

Unfortunately, the adoption of "Agile" in these organizations has reinforced the culture of "We don't need no stinking requirements! We can draw an architecture on a whiteboard in an afternoon", resulting in systems where you really can't say anything intelligent about how long it will take to complete them, because you have no fscking idea what "complete" actually is.

And this -should not be a revelation-, at least to anyone who has read "Mythical Man-Month," which will be 40 years old next year. https://en.wikipedia.org/wiki/...

Thank God I'm getting ready to retire.

Re:Technical People

[LordLucless]

Sadly this is true, but it shouldn't be. Technical people should have the professionalism to analyse requirements and check that the requirements fit the purpose.

Most I know do. The problem is that they're not sufficiently expert in the domain (in this case, health care) to determine the purpose, and the purpose the client gave them is wrong.

Specs aren't just some bureaucratic hoop that needs to be jumped through to get a developer to sit down and code, and they're not something a developer can just wing, and get right anyway, because they already knew what they were and were just being anal about getting you to write down.

They are important, and if they're not done properly, the dev will likely spend a lot of time doing the wrong thing correctly, and you will be billed for it.

Re:Technical People

[Anonymous Coward]

Unfortunately the way of the world is that technical people would be quickly shuffled out of the way by sales and marketing if they started to reduce revenue by telling a customer what they really wanted instead of what the spec says.

Disclaimer: I'm a software engineering contractor that works on contracts for the federal government.

A solid majority of the contractors (the grunts doing the work) I've worked for/with in my career want to get the job done and do it well. Sales/marketing has a say at contract award and for mods, but during the actual work we rarely, if ever, hear from them or take guidance from them. The people commissioning the work (the government) usually have no clue what they want and, if presented with multiple solutions of varying risk and value, they still have no idea how to make a decision. The most altruistic contractor still, at the end of the day, needs to know loosely what the success criteria are...the government half the time has vehement disagreement about that among themselves and never comes to a unified decision.

The GAO's report is exemplar of what I've experienced...the government has no clue what requirements are or should be, how to execute, how to manage a contract. My contracts have routinely consisted of us contractors drafting requirements and handing them over to the government, only to have them ask us if they were sufficient and would accomplish the (loosely defined) task, then sign them, hand them off to contracts and they appear on our desk weeks down the line modified by contracts to be 1) more generic, or 2) incorrect. The government oversight at the program manager level is almost entirely a rubber stamping process.

Re:Technical People

[oneandoneis2]

"Lorem ipsum" is industry standard "filler" text for incomplete web pages - typically used to show clients what a page will look like when it has some useful content.

Not that it isn't appalling that it's appearing on a page in production, but it isn't "random Latin" - there are even browser extensions to make it easy to C&P for you.

Re:Technical People

[Anonymous Coward]

sales & marketing (not entirely unfair, they are typically huge scumbags)

I'm a web developer who works in the marketing department of a large organization. The people in my department are smart professionals who are tasked with keeping the organization on-message and professional in its communications with the outside world. This is an immensely difficult herding-cats kind of job because so many different departments and individuals are communicating with the public every day, and many of them do so in a way that unnecessarily casts the organization in a negative light. Sometimes it's just a matter of professionalism (poor grammar/spelling, rudeness, childishness), and other times it's because they're uninformed and telling people things that simply aren't true, which ends up confusing everyone.

Our department has a broader and deeper understanding of this organization than anyone else here, including the top leadership. We're the ones who have to continuously remind everyone else of the organization's guiding principles and priorities. And every time someone sends out yet another bulk email to 20,000 people in pink Comic Sans containing information that was no longer accurate as of 2007, we're the ones who have to beg them, yet again, to run their communications by us before they send them out. In fact, how about we just start sending your materials out for you? We'd be happy to. No, really, it would be our pleasure.

Contrary to common opinion, that's what a lot of marketing jobs are really like. Maybe some marketing people are scumbags, but not the ones I work with.

in other words

[ganjadude]

it was a giant clusterfuck like many people on both the left and right were claiming way before launch. the site was NOT ready for prime time (the back end still is not 100%) and it never should have been launched when it was.

also, water is wet

Re:in other words

[ganjadude]

congress tried to delay it before the government shutdown remember? obama would not budge, causing the government shutdown.

after the shutdown the site launched, and as expected obama changed his mind and delayed implementation anyway

so the reason for the shutdown was that the democrats did not want a delay and wouldnt budge. then when the site launches and makes them look bad, the implement the delay anyway... yet they still blame congress for the shutdown. and based on your comment it seems some americans are still dumb enough to believe it

Re:in other words

[Anonymous Coward]

Pull your head out of your ass. Seriously. You've got the president going on record of saying he is not going to negotiate and actually claiming he's going to use executive order to bypass the limitations of power spelled out in the constitution during the state of the union, and you've got congress throwing up road blocks to do everything to basically try to stop Obama. I'm going to throw a shocker at you. You might want to sit down for this. I'm a conservative, and I'm happy with congress. The dems in the first two years when they had full control shot out of control. They did whatever they want with the justification of "well, we're the majority, we can do what we want". I'm sorry, that's not how the world works. If you had a 90% majority, then yes, but you don't. You have a 51% or 52% majority. When that's your majority, you need to consider what the other side wants, you don't get your way all the time.

What the dems pulled in the first two years of Obamas presidency set the stage for what's happening now. I voted in people to put the brakes on you guys, and they're doing exactly what I wanted them to. How about this, how about you sit back and say "hey, conservatives, okay, we need to live in this world together, how about we sit down and try to find a solution we can both tolerate". And please note that tolerate doesn't mean like. Remember, we have both parties right now refusing to negotiate. It's not one side or the other. The presidents own words can be quoted to attest to this.

And also note, I'm not claiming conservatives are blameless here either. But you just threw out a load of tripe putting all blame on one side, when both sides stink so bad they should all be thrown in the garbage.

Re:I'm so glad

[Chrisq]

I don't work for a company that made the mistake of getting involved in that nightmare.

I'm pretty sure that a lot of companies are doing just fine out of it - paid to deliver the wrong thing then paid to deliver what the government should have specified in the first place.

Let's be absolutely clear

[argStyopa]

The key takeaway from the report is that nobody will be personally held to blame for the incompetence (at best; corruption and nepotism at worst) of the process and end result.

No punishments or consequences, all around!

better summary

[slashmydots]

Don't hire people who have failed multiple projects in the past just because they were friends of the Obama campaign. At least that's what my finding determined.

Re:what's the difference

[ganjadude]

the difference is that if its a private company doing it, we are not all paying for it. I dont care if a private company wastes a billion dollars, that has nothing to do with me and the rest of america. but when the government does it, it becomes an issue for all of us

Re:I'm so glad

[StikyPad]

It's actually relatively common for custom software to experience feature and scope creep. The source of creep is split between design by committee and leadership changes. When new leadership comes in, the vision almost always changes, and when new stakeholders are added, they pollute the water with their own special interests.

It's arguably the role of developers (or at least business analysts) to push back against ridiculous requirements, and some do, but they're not properly incentivized, since they work for the contractor. BAs should be working for the government, not the contractors. Ideally, one person with software development design and management experience and a clear vision should be in charge of the project. Unfortunately, it's almost always someone with more generalized management experience who doesn't know the difference between HTML and CSS, and comes up with new "great ideas" on the fly.

At any rate, the problem isn't limited to government software -- I've seen the same thing in commercial business software, especially "customizable" software. I'm looking at you, mortgage and scientific industries. We get a little more upset because we fund government software through taxes -- we feel like it's our money -- but we honestly fund almost all poorly designed software, even if it's rolled into our mortgages. It's just less transparent.

Re:Why dont we

[ka9dgx]

Because 10% of a working system can't be measured. Even a 100% completed to spec system is worthless until it has actually been used for a while... when it will prove to need about 100% more work.

Most software projects fail, unlike construction, etc... engineering can't be applied.

F-35 Joint Strike Fighter

[tekrat]

Really, we want to complain about a website that cost a Billion? This is the United States Government, full of waste, fraud, no-bid contracts, and shit spread out out over every state so that ever senator and congressman has his slice of the taxpayer slush fund.

Witness the F-35 Joint Strike Fighter, an aircraft nobody needs, trying to fill too many roles, and was supposed to save our armed services money by having one plane replace many planes.

Except it's billions over budget, still doesn't work (and might never work), and is expected to cost more than a Trillion dollars before all is said and done.

Meanwhile the aircraft is being usurped by drones, which are cheaper, easier to deploy, and may fill all the roles we'd ever need this crazy ass jet for. And we're trying so hard to make it stealthy, meanwhile as pointed out in a slashot article a few weeks back, long wave radar will find the plane just fine.

And yet the Pentagon continues to shovel more money into the project because -- guess what, there's no "plan B". This is the people we depend upon to strategize for us in times of war, and they have absolutley no fall-back plan. Brilliant.