Slashdot Mirror
Lessons From the Healthcare.gov Fiasco
Nerval's Lobster writes "In theory, the federal government's Health Insurance Marketplace was supposed to make things easy for anyone in the market for health insurance. But fourteen days after the Website made its debut, the online initiative—an integral part of the Obama administration's Affordable Care Act—has metastasized into a disaster. Despite costing $400 million (so far) and employing an army of experienced IT contractors (such as Booz Allen Hamilton and CGI Group), the Website is prone to glitches and frequent crashes, frustrating many of those seeking to sign up for a health-insurance policy. Unless you're the head of a major federal agency or a huge company launching an online initiative targeted at millions of users, it's unlikely you'll be the one responsible for a project (and problems) on the scale of the Health Insurance Marketplace. Nonetheless, the debacle offers some handy lessons in project management for Websites and portals of any size: know your IT specifications (federal contractors reportedly didn't receive theirs until a few months ago), choose management capable of recognizing the problems that arise (management of Healthcare.gov was entrusted to the Medicare and Medicaid agency, which didn't have the technical chops), roll out small if possible, and test, test, test. The Health Insurance Marketplace fiasco speaks to an unfortunate truth about Web development: even when an entity (whether public or private, corporation or federal government) has keen minds and millions of dollars at its disposal, forgetting or mishandling the basics of successful Web construction can lead to embarrassing problems."
There's a cream for that.
But we can't tell yet if your insurance will cover it.
"What in the name of Fats Waller is that?"
"A four-foot prune."
Remember that this is only for people that live in states that tried to stall off the inevitable. I live in Kentucky and despite being a pretty red state we have a Democratic governor and he saw the writing on the wall. Rather than try and delay and delay it we have our own. Numerous other states did the same thing. I haven't heard anything about ours being down.
And you have to realize that not everyone on the team has the same goals.
How much do the contractor companies get paid for overtime or change requests?
When I'm a contractor I will tell you what problems there could be that I can see. But if you tell me to do it your way I'll do it your way and collect my check.
Part of the problem is the usual problems with large-scale IT projects: it's not until you're well into it that you really get a grasp of what's involved. Nothing government-specific there, that plagues all large IT projects in private industry. Part of the problem, though, lies exactly in the fact that contractors were used. Contractors are mercenaries. They're here to deliver this project, and once they get their paycheck they're on to other work. They won't be around to deal with the fall-out and maintenance headaches from their work, and they don't have any vested interest in the quality of their work as long as it's good enough to pass review and get their payment check cut. In fact, poor quality is actually an opportunity to get paid twice since fixing the problems is a new project. Full-time permanent employees may not be as efficient as contractors, but on the other hand they've got a vested interest in making sure the system doesn't create any more problems than necessary because they know they're the ones who're going to have to clean up the messes. Long-term employees also have a better grasp of what's already involved in the current system, which translates directly into a better grasp of what the new system will need to do. They're less likely to miss major complications because they already have to deal with them.
Part of the problem with contractors is also the fact that large organizations like governments limit themselves to Tier 1 contractors. And there aren't a lot of those. So it rapidly becomes a situation where the Tier 1 contractors aren't really concerned about quality and results, because they know their customers will by policy refuse to consider any alternatives outside a small set and those others aren't any better about quality. If the government switches from contractor A to B, that means B can't take on another customer who takes their business to A (because A and B are the only Tier 1 firms and the customer can't consider anyone who isn't a Tier 1 firm) and it's a net wash for A.
This
made me laugh.
I've been in the unfortunate situation of working for a government agency when Booze Allen Hamilton came in to help make changes and improve things. They did much of the former and none of the latter.
Typically, dealing with whoever was going to actually use the process they were changing was something the Boozers did just to check off an item on a list. They did not listen to users because they assumed government employees were all idiots and could tell them nothing they really needed to know about the processes they were about to change.
Personally, if I were going to change business processes that had been in place for decades I'd want to talk to the people who work the current processes and find out how they work before I started trying to think up better ways of doing things. BAH never did that. They brought in workers for planning sessions, listened for a couple of days, then distilled the results of those discussions into a document of findings that was obviously written before the research ever started and contained exactly zero input from the field workers who truly understood job requirements.
Boozers, in my organization, were almost universally so convinced that their shit didn't stink that they were worse than useless. In the course of years of contacts with them, I met exactly ONE who listened, learned, and improved things.
Based on those past experiences, I can only surmise that the folks responsible for this current fiasco simply said "Oh, we don't need to talk to anyone from the government about how they run web sites that stand up to incredible traffic swings. We know what we're doing."
And some idiot government executives trusted them.
I don't know who to be more disgusted with.
I don't know about you, but does the site really need links and JS from ad sites (like doubleclick, chartbeat), YouTube, and Facebook, as well as whatever googletagmanager and optimizely provide - as noticed by what I had to temporarily allow in NoScript - to simply make the site work to, you know, helps people get access to healthcare insurance?
It must have been something you assimilated. . . .
The reason for the mandate (and for the original single-payer system) is that currently the cost of health care for the uninsured is hidden in the "uncollectable debt" category in the hospital's accounts receivable. It's all the bills for ER visits and emergency care for people who can't pay. I was taught a basic rule back in high school business classes: you can't manage costs until you've got them laid out where you can see them. The idea was to get all health care being paid for and accounted for so we can see where the money's going and do something about the areas where it's costing more than it should. It was also to help with shifting the costs from expensive emergency care to much cheaper preventative care, the idea being that when people know they're covered by insurance they're more likely to go to the doctor before things get critical instead of putting it off and hoping they get better so they don't get nailed with a doctor's bill and ending up at the ER in critical condition. If you have no insurance the bill's going to be a killer either way so it makes sense to go for the chance to avoid it, whereas if you do have insurance the bill won't kill you either way so why wait and suffer more than you have to?
This happens every time a major new internet service is launched. And it _always_ will. See, here's the problem: at launch everyone is interested and wants in. After a few weeks/months the interest dies off and the site hits a BAU point. So if you're designing one of these sites you're stuck either:
a. Spending billions on infrastructure for 3 months tops of high volume and then getting ripped to shreds in the press for 'wasting' all that money. or...
b. Taking your lumps up front and waiting a few months for people to forget about it.
The guys running healthcare.gov opted for 'b.', and I would too. The kinds of people that just want to say bad things about the ACA would have a field day with 'a.', with 'b.' they'll have to acknowledge (or at least ignore) the fact that in a few months it'll be working more or less as intended.
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
"...but I seriously had no idea it would be this bad."
How would you design a Healthcare Extortion Racket?
"New York State's healthcare plans range from Fidelis Care's 'Bronze' plan at $810.84 per month to $2554.71 per month for something I didn't bother to look up because if I had $2500+ a month to spend on doctors, I'd buy a doctor and have him/her live with me and dole out pills like I was Michael Jackson. The deductibles - the amount you pay out of pocket every year before you the insurer has to give you anything at all - are outrageously high. Fidelis Care Bronze has a $3000/year deductible per person. I'm in pretty good health; it's a rare year I spend that much on doctors. After the $3000/year deductible, they pay 50% of your bills. So if you rack up $5000/year in medical bills, you pay $4000 and they pay $1000. Pretty damned crappy."
Repeat, from my JE.
"Flyin' in just a sweet place,
Never been known to fail..."
It's pushing 20 years since I first saw an academic study showing that IT project failure probability increases dramatically - the latest was 2005:
The Challenges of Large-Scale IT Projects
You're darn right I won't be put in charge of such - not without a gun to my head. I'd want to de-scale anything down to a size where you could reasonably spec and test it. As the article says, "test, test, test". A formative experience in my programming was FORTH, a language that strongly rewards small incremental experiments, compiling as you go, building from small functions up to large ones. I'm not saying use FORTH, but the philosophy of getting the basics working and building up has really worked for me for a whole career.
By contrast, all the large-scale projects I've worked on have all taken a philosophy like building a skyscraper or 747 - no one person can comprehend it, design everything before the first screwdriver is picked up, so the design process goes on for months and years, etc. And then you have "crunch time" from then on as the fond beliefs of the design team smack into reality, and the specs are proven to not match needs. Incremental building and testing tends to reveal these problems.
The fear that drives these philosophies is that you'll have the thing mostly built...and discover it doesn't meet every need and can't without some huge rebuild, because you didn't think of everything up front. Rather like an old system that's been patched to death and has to be tossed because it just can't keep up with changes. I think the fear exaggerated, particularly if the design-build team is at least roughly aware of the whole project dimensions.
The advantage of more-incremental projects that are never large because you take one part at a time is you develop in priority order. The 80/20 rule suggests that 80% of the clients will want about 20% of the options available - so get 20% of the offering working, and working well, first.
Canada has this story of medical records: http://news.slashdot.org/story/09/10/10/0124227/open-source-could-have-saved-ontario-hundreds-of-millions /. covered it, "open source" would have saved 95% of project costs, but I think it was also about the open-source development was in small increments, no large projects.
As
Clearly this is all because of {current President} and all the rest of the {President's affiliated party}. If only this were done by {opposing party to that of the President} instead none of this would have happpened!
What production website do you know of where development stopped and the product had no issues?
Development is an ongoing process that goes up to and beyond the day of product release. There will always be security holes to patch, bugs to fix, changes to be implemented, new features to be added.
If you think you can release a product on day x and lay off / fire / furlough all the developers the same day and have a good product you are part of the problem not part of the solution.
Considering testing was slated to BEGIN the day before launch, I doubt it.
The contractors were too busy designing all of the "Due to the government shutdown, this website is closed" websites for all of the other government sites, so they didn't have enough time to work on the launch of the new site.
No, wealth is unaffected by inflation. Wealth is not a stack of dollar bills. You must me invested in the means of production to have wealth, and the value of that is determined by what's produced, not the currency in use.
Hyper-inflation destroys savings, not wealth. Usually, hyper-inflation also destroys economies and governments. And, of course, it would be hyper inflation: with no practical limit on how much the government could spend, it would try to spend infinity dollars on pork barrel projects and outright checks mailed to supporters.
I doubt people would but precious metals, though, there are several stable national currencies, easier to just us Canadian dollars or Swiss francs or whatever, if it came to that.
Socialism: a lie told by totalitarians and believed by fools.
I wouldn't be so sure.
In my experience of building e-commerce sites (over roughly five years) the actual *building* of the site isn't the difficult part. I've taken a barebones install of Magento (or Prestashop, etc) and themed the front-end, by myself, in just two/three weeks. Looking at this site, I can't imagine the front-end would take any longer.
The proverbial iceberg, though, is what you're looking at. The bits that take the most time are all the logistics bits like shipping, payment processing, which customers can purchase what, how are discounts handled, tiered pricing, product entry, admin training, etc. I would guess that 90% of my conversations with clients over the years involve some logistics bit, not whether the buttons on the checkout page are the correct color of blue.
And then to top all that off, you have the infrastructure to worry about. You aren't necessarily dealing with a web server and that's it. You might have a cluster of web servers that need to talk to a cluster of SOLR servers. You might have to implement solutions for payment processing servers.
In the end, these items all take a great amount of time not because of how complex they are to implement, but instead it has everything to do with the *people* that are organizing this information. Hell, can you just imagine the nightmare it must have been to get all the insurance companies to provide all their data/plans in a standardized format so they could be integrated to the store front?
In the end, though not unexpectedly, they ran out of time and testing was shat upon. Every relatively complex site I have ever built or worked on has had testing shat upon. Now that I have just a single site that I develop and work on, testing happens all the time since I am my own boss as far as deciding what I need to work on. For every other project out there where the developers aren't the ones that even have a say in what areas are focused on, testing will always be a second-class citizen.
Someone flopped a steamer in the gene pool.
So the logical thing would have been to install those fences and station armed guards every evening after the memorial was vandalized. Not all day a couple months later to keep daily visitors out.
Keep defending it if you want, but the people of this country know when someone is acting like a spoiled child, taking his toys and pouting in the corner. This has nothing at all to do with the shutdown, and everything to do with an ego.
If you think I voted for Trump because of this post, you're wrong. I voted for Dr. Jill Stein of the Green Party. Again.
Well, no. The Dot Bomb was caused by wild irrational exuberance. I have yet to meet a government bureaucracy that was characterized as having âoewild irrational exuberanceâ.
And I think there is an important difference. If a company has poor customer relations people will go elsewhere or start up their company. The old companies will go bankrupt. The amount of damage is limited.
Government bureaucracies are different. If they fail they don't go out of business. Often another layer of regulation and bureaucracy are laid on top of the old so the whole thing grows. (I also think it ties to incentives and who chooses to work for government. In business risk generally have huge upsides with limited downsides. In government that is reversed. Risk taking is meet with limited upsides and serious downsides.)
It's a website that needs to be able to handle 3million visitors per day, with the majority of them being signups, or at least hitting the calculator. That's a lot of deep hits that can't be cached.
Then, add on a back-end that has to talk to insurance companies. These guys still have a tonne of Cobol code running around. There's nothing wrong with that (Seinfeld!), but I think it might indicate that their systems aren't necessarily built for online, real-time querying.
To recap, it is a multi-tier system:
1) Front end, performing user signup, and calculator.
2) Back end database. HIPA compliant, Sarbanes-Oxley compliant and able to deal with 100m customer records.
3) Feeds to remote systems, also HIPA compliant, Sarbanes-Oxley and other stuff.
So, you've got something that looks a lot like twitter (the back-end links), only more expensive because it needs to be Capital S secure, along with something that looks like an insurance company (the middle tier) and finally something that looks like a dot-com (front end calculator).
That's already a lot of hardware and software. "Free" open source doesn't actually save a lot of money here, since most of the money is in support (over 1/2 the 5year cost!). Now, triple it do deal with hot site failover, backups and other various disaster recovery plans.
Although they've had 3 years to get the system complete, the software was probably only developed in the last 10-12 months (at most). The rest of the time would have been spent in getting agreement on the data exchange formats with the insurance companies, deciding on a vendor to use for each part, and standing up an internal team to manage it. Then add in several parties involved playing schedule chicken with Congress, hoping for the whole thing to either be delayed or scrapped. Fun!
Finally, they went for a nationwide rollout for political reasons, which was guaranteed to result in peak traffic on day 0.