Slashdot Mirror
Administration Admits Obamacare Website Stinks
Hugh Pickens DOT Com writes "The WSJ reports that six days into the launch of insurance marketplaces created by the new health-care law, the federal government finally acknowledged that design and software problems have kept customers from applying online for coverage. The website is troubled by coding problems and flaws in the architecture of the system, according to insurance-industry advisers, technical experts and people close to the development of the marketplace. Information technology experts who examined the healthcare.gov website at the request of The Wall Street Journal say the site appeared to be built on a sloppy software foundation and five outside technology experts interviewed by Reuters say they believe flaws in system architecture, not traffic alone, contribute to the problems. One possible cause of the problems is that hitting 'apply' on HealthCare.gov causes 92 separate files, plug-ins and other mammoth swarms of data to stream between the user's computer and the servers powering the government website, says Matthew Hancock, an independent expert in website design. He was able to track the files being requested through a feature in the Firefox browser. Of the 92 he found, 56 were JavaScript files... 'They set up the website in such a way that too many requests to the server arrived at the same time,' says Hancock adding that because so much traffic was going back and forth between the users' computers and the server hosting the government website, it was as if the system was attacking itself. The delays come three months after the Government Accountability Office said a smooth and timely rollout could not be guaranteed because the online system was not fully completed or tested. 'If there's not a general trend of improvement in the next 72 hours of use in this is system then it would indicate the problems they're dealing with are more deep seated and not an easy fix,' says Jay Dunlap, senior vice president of health care technology company EXL."
So the story here is that a large team of software developers with no demonstrated experience in developing, testing, performing quality assurance for, and administering large scale enterprise application deployments get a federal contract and botches it horribly. Color me shocked.
I've been working in development and architecture roles for fifteen years, and have seen exactly the same pattern on a variety of scales over and over again. I've seen a number of rather large infrastructure development projects that worked out very well too, but none of those were public sector projects.
Just remember that the folks responsible for this mess are certainly still taking paychecks while an enormous number of government workers are suffering due to the inability of our Congress to do its job. Good times, huh?
Write failed: Broken pipe
I'm confused, I thought that nobody wanted obamacare?
"The WSJ reports that six days into the launch of insurance marketplaces created by the new health-care law, the federal government finally acknowledged that design and software problems have kept customers from applying online for coverage."
What software platform does the software run on ?
I think this problem has less to do with the platform and more to do with the fact that this is what you get when you take the lowest bid without doing some basic research on the competence of the bidder. I mean 92 files per 'Apply'? Seriously? And they rolled it out after the Government Accountability Office warned that insufficient testing had been done? This mess says something about the people running the project. It seems to me that those three months could have been well spent hiring software testing contractors to do some load testing although one gets the feeling from the descriptions that team working on this system were scrambling so madly to get it working by their deadline that there would probably not have been any time to fix any except the very worst the bugs the contractors would have found.
Only to idiots, are orders laws.
-- Henning von Tresckow
Doesn't matter. It's a government job, and everyone involved makes more money if it's a ten-year debacle than if it actually works.
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
Healthcare.gov problems are real. But asking for opinions from people who have a dog in the fight is probably less than ideal. When you ask the likes of Wall Street Journal (Rupert Murdoch's conservative rag) or healthcare technology company EXL (sour that they did not get the contract), you'll get answers that are entirely predictable.
Why is the website a clusterF? Several reasons come to mind.
1. It is a 1.0 product.
2. It is a government project, what do you expect?
3. The states who setup smaller (in comparison) exchanges had similar problems. My state of OR paid Oracle about $50,000,000 for a much simpler setup where you cannot buy anything, but can only view plans on offer. And even that did not work for first few days.
4. The developers were stupid and did not anticipate the traffic they got. Even engineering oriented companies like Google often make that mistake. If you have ever tried registering for Google I/O you would know what I am talking about.
5. Obama's coding skills are simply not up to snuff.
Team Red would like you to think that the govt. has all of a sudden become very inefficient under Obama's presidency. And under their guy Bush, it was a model of transparency and efficiency.
> Silly question, but... what happens when you want to apply and you don't have a computer ?
Obamacare by phone: 800-318-2596
Oregon did just that. About $50mil later they had a website that did not work for the first few days. And it is a view-only site to begin with.
Giving lots of money to a large company is no guarantee of success.
Let's examine an HTTP request for a rather beefy portion of the JavaScript in question from healthcare.gov:
They're not even bothering to set the HTTP Cache-Control, Proxy-*, or Expires headers on this content, which will most assuredly limit intermediary proxy and client caching. To say this is amateur hour would be a gross exaggeration of the skills being fielded by these developers.
Much larger issues undoubtedly exist in their backend infrastructure. Given the shit I've seen in this area, I could probably spend the next hour making educated guesses about how badly they've fucked up in various regards, spend another hour partially validating those guesses, and wind up just saying "yup, they're idiots." Instead, I think I'll go to bed now. I have work in the morning.
Write failed: Broken pipe
Mongo is webscale.
The USA is frighteningly-close to tumbling into full totalitarianism.
You were doing so well - and then you threw in this bit of unsupported insanity.
Indeed. Remember that Bush/Cheney failed experiment of outsourcing the Iraq War to private companies - companies that brought in untrained "experts" to interrogate prisoners, private security companies to police the streets like the Blackwater employees who killed 17 civilians in Nissor Square, Bahgdad thinking they were being fired upon, or the Halliburton contractor who improperly installed water pumps that killed over a dozen American soldiers while they were showering. Libertarians and anti-government conservatives that complain that government never works while living in a country in which quality of life is almost purely dependent on government programs - like freeways, municipal transportation, clean air, water systems, waste disposal, the internet, police departments, etc, etc, etc - should really just move to Afghanistan.
I'm a bit surprised that we seem to accept the "Obamacare" nomenclature. Can we at least try to be objective? http://www.prosebeforehos.com/video-of-the-day/10/06/obamacare-versus-affordable-care-act/ http://tv.msnbc.com/2013/09/27/poll-more-oppose-obamacare-than-affordable-care-act/
work in progress
That's exactly the quality you get when you outsource to Indian programmers. We've had a decade to evaluate the outsourcing debacle...haven't we learned any lessons from it?
Mongo loves candy...
Do not look at laser with remaining good eye.
So now Obama can agree to a later start of Obamacare without losing his face: He'll not give in to the Republicans, but just react to deficiencies in the technology.
To add insult to injury, the administration decided to take down the Amber Alerts website, blaming the shutdown, but Michelle Obama's "Let's Move" website is still up. They shut down the PX at Andrews AFB and the WW2 Memorial on the National Mall to WW2 vets, but the golf course at Andrews AFB, which Obama likes, is still open, as is the one at Camp David. Funny what this administration considers "essential".
For this administration it's about not compromising and punishing the American people for supporting their opposition. The pain they intentionally inflict they hope will convince most people to force the opposition to give in. A Park Services Ranger was quoted as saying they were told to make life as painful as possible for people.
"Tell your Senator/Representatives to cave or this kitten (or abducted child that won't show up on the shut-down Amber Alert website) gets it."
1. Nudge
2. Shove
3. Shoot
They are past "Nudge" and are now well into "Shove"...with scattered, mostly kept low-key (for now), but increasingly-numerous incidents where "Shoot" is starting to be employed.
The USA is frighteningly-close to tumbling into full totalitarianism.
Strat
Seriously? You're going to reference The Examiner for the park ranger quote? Come on.
For the rest Reuters has a good explanation of why parts of the government are hit by the shutdown and other parts continue unaffected, the explanation being that the parts that get funding from Congress stop and those and which are funded otherwise continue to function. In the case of the Andrews AFB golf course, for example, it's funded by user fees and is not reliant upon Congress for budget.
Source: http://www.bloomberg.com/news/2013-10-03/troops-forage-for-food-while-golfers-play-on-in-shutdown.html
But hell...don't let details get in the way of your rant...
blindly antisocialist = antisocial
Mongo just pawn in game of life.
No sig today...
Not really.
They're built by lowest bidders Serco and QSS Inc. Neither an American company.
If they had decided to hire Americans to do this job, they would have had a very large pool of qualified and skilled workers from which to choose.
They're using their grammar skills there.
Consider Healthcare.gov as an Engineering project. Under .gov procurement rules. . .
The law: an ~1800-page CONOPS document.
The 10K+ pages of accompanying regulations ? User requirements.
So. . .CONOPS passes approval, User reqs start getting gathered. Someone writes an RFP and puts it out for bid. Given typical Fed procurement requirements, that's 9 months to a year before contract award. PPACA passed in March 2010, so we're probably at March 2011 now.
Winner ramps up, develops a Performance Spec and Initial Design, and starts procurement of infrastructure required. Another 6 months. Sept, 2011 now.
Infrastructure stand-up and development begins. Likely another 3 months. It's 2012 now. Standard development and monitoring/audits. Pilot of basic site for Insurance Exchange, though reviews and changes. 6 months min, 9 months likely, Sept 2012.
In the next year, you need to finalize, get the integration between multiple .gov sites and agencies hashed out and tuned, and THEN go to useability, security, and scaling tests. In ANY .gov program, that's 2 years, minimum.
Which means, the first REALISTIC date for Exchange eligibility would have been October 2014. But the lawyers and politicians didn't bother asking the ENGINEERS how long it would take, they never do.
And **THAT**, is my best estimate of what went on and what is going wrong. . .
You, are a fucking moron.
He didn't shut down the ocean.
http://www.politifact.com/florida/statements/2013/oct/07/tweets/did-obama-shut-down-ocean-part-shutdown/
And he didn't shut down the Amber Alert system. The Amber Alert system is a private non-profit entity at the federal level so he couldn't shut it down even if he wanted to.
http://www.politifact.com/truth-o-meter/statements/2013/oct/07/tweets/tweets-and-bloggers-say-obama-used-shutdown-close-/
I don't know how you could ever post something from Breitbart with a straight face.
It has the last-modified header and an Etag. Expires and cache-control are unnecessary. Contrary to popular web developer belief.
http://redbot.org/?descend=True&uri=https://www.healthcare.gov/&req_hdr=Referer%3Ahttps://healthcare.gov/
http://redbot.org/?uri=https://assets.healthcare.gov/global/js/lib/jquery-1.8.2.js&req_hdr=Referer%3Ahttps://healthcare.gov/
HTTP/1.1 200 OK
Server: Apache
ETag: "cfa9051cc0b05eb519f1e16b2a6645d7:1370524513"
Last-Modified: Thu, 23 May 2013 15:59:12 GMT
Accept-Ranges: bytes
Content-Type: application/x-javascript
Vary: Accept-Encoding
Content-Encoding: gzip
Date: Tue, 08 Oct 2013 11:58:37 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Connection: Transfer-Encoding
General
The server's clock is correct.
Content Negotiation
The resource doesn't send Vary consistently.
The ETag doesn't change between negotiated representations.
Content negotiation for gzip compression is supported, saving 64%.
Caching
The resource last changed 137 days 19 hr ago.
This response allows all caches to store it.
This response allows a cache to assign its own freshness lifetime.
Validation
If-Modified-Since conditional requests are supported.
An If-None-Match conditional request returned the full content unchanged.
Partial Content
A ranged request returned partial content, but it was incorrect.
I am certainly NOT a proponent of out-sourcing (I will not debate my reasons here). However, let's put the blame squarely where it belongs - on the accepted process of hiring the lowest bidder with no vested interest in getting it right vs one where getting it right would have great impact on the users.
If this work was being done by Americans who actually need to rely on the ACA for their health care coverage, you can bet your ass that it would have been done right - the first time. And, those who are involved can say it was an American success story. Instead, we now have another reason for it's opponents to call the whole program a failure.
Brilliant.
There's a thing called HTTP 1.0, and in it there's a feature called Connection: Keep-Alive. It doesn't spawn a new TCP connection for each of those 56 javascript files. Only one TCP connection per (sub)domain is made when Keep-Alive is in use. This was such a nice feature that in HTTP 1.1, all connections are considered persistent "keep-alive" unless you write Connection: Close. From a network standpoint a few extra lines of HTTP headers between each script isn't going to matter, and if it's cached and/or co-located properly (eg: via Akamai), it actually does matter, since those requests are going to be served from the caches efficiently.
However, the biggest problem is that HTTP is fucking dumb. No, really, it's dumb. Not that it's designers were dumb, just that it's evolved over the years and security was never part of the design. For one, there is no such thing as a "Session". In this day and Age of Information that's ludicrous! Say you use a session cookie to validate every single request for every single resource is valid... because that's what you have to do, then EVERY COOKIE gets sent to the server EVERY TIME you make a request. It's so much face palm, I can feel the back of my skull.
On the security standpoint, neither HTTP or HTML really knows how to actually work with encryption. That happens in TLS. What a fucking crock of shit. HTTPS means you can't cache anything. Most of the files being served are NOT dynamic, but STATIC files. However, since HTTP/HTML are so fucking dumb they can't even provide a simple hash, then you can't trust mixed content. If in addition to the URL of a static resource, you could also include a known hash:
<img src="..." digest="d8b09c45b522e34d81ac9eed95f922c7028e7fb2; type=hex/SHA-1">
Then the browser could hash the unsecured (cache-able) resource as it's pulling it in at the behest of the secured dynamic (uncatchable) page, and verify that the requested unsecured content wasn't tampered with in transit so it wouldn't be a security issue and we could actually FUCKING USE SECURITY EFFICIENTLY, grrr. Especially if you could specify a few bits of salt with the hashes...
<img src="..." hmac="WkRoaU1EbGpORFZpTlRJeVpUTQo=, TlRJeVpUTTBaRGd4WVdNNVpRbwo=; type=base64/SHA-1">
But, no, that doesn't exist. No HTTPS content is cached. Apparently I'm the only one on the planet not drinking the damn cool-aide. The web is bloated and retarded, it needs to die. Long live the Internet, but fuck the web. It took HALF the age of the Internet just to get from HTTP 4.01 to HTML 5... Over a Decade, and this shit still isn't in the spec. Don't hold your damn breath for next version, or for anyone with a fucking clue how things should work to propose sane changes. Even Google with SPDY is just exacerbating the issue with bandaids over the inefficiencies of HTTP.
TL;DR: Yeah, it's a shitty website / backend design, but primarily it's because HTTP/HTML is just fucking retarded.
This is just one of those things that the government really doesn't do all that well. Private organizations live and die by their profit margin, so they make damn sure shit works and it works affordably.
I cannot let this comment pass. Sorry, but anyone who's worked for a large corporate beauracracy knows this is nonsense. They are just as large, Byzantine, and wasteful. That's simply how large human organizations function.
Natural != (nontoxic || beneficial)