Slashdot Mirror
Administration Admits Obamacare Website Stinks
Hugh Pickens DOT Com writes "The WSJ reports that six days into the launch of insurance marketplaces created by the new health-care law, the federal government finally acknowledged that design and software problems have kept customers from applying online for coverage. The website is troubled by coding problems and flaws in the architecture of the system, according to insurance-industry advisers, technical experts and people close to the development of the marketplace. Information technology experts who examined the healthcare.gov website at the request of The Wall Street Journal say the site appeared to be built on a sloppy software foundation and five outside technology experts interviewed by Reuters say they believe flaws in system architecture, not traffic alone, contribute to the problems. One possible cause of the problems is that hitting 'apply' on HealthCare.gov causes 92 separate files, plug-ins and other mammoth swarms of data to stream between the user's computer and the servers powering the government website, says Matthew Hancock, an independent expert in website design. He was able to track the files being requested through a feature in the Firefox browser. Of the 92 he found, 56 were JavaScript files... 'They set up the website in such a way that too many requests to the server arrived at the same time,' says Hancock adding that because so much traffic was going back and forth between the users' computers and the server hosting the government website, it was as if the system was attacking itself. The delays come three months after the Government Accountability Office said a smooth and timely rollout could not be guaranteed because the online system was not fully completed or tested. 'If there's not a general trend of improvement in the next 72 hours of use in this is system then it would indicate the problems they're dealing with are more deep seated and not an easy fix,' says Jay Dunlap, senior vice president of health care technology company EXL."
So, you're saying that the web site is a proper government software project? ;-p
Ezekiel 23:20
So the story here is that a large team of software developers with no demonstrated experience in developing, testing, performing quality assurance for, and administering large scale enterprise application deployments get a federal contract and botches it horribly. Color me shocked.
I've been working in development and architecture roles for fifteen years, and have seen exactly the same pattern on a variety of scales over and over again. I've seen a number of rather large infrastructure development projects that worked out very well too, but none of those were public sector projects.
Just remember that the folks responsible for this mess are certainly still taking paychecks while an enormous number of government workers are suffering due to the inability of our Congress to do its job. Good times, huh?
Write failed: Broken pipe
Silly question, but... what happens when you want to apply and you don't have a computer ? Surely, by definition, a sizable portion of the population that requires Obamacare doesn't necessarily have the means to have a computer or an internet connection.
And no, "anybody has a computer these days" is not an answer. I know plenty of people who don't have enough to feed themselves, let alone buy a computer - let alone one that's recent enough to cope with plugins that invariably tell you "your operating system / browser is not supported anymore, please upgrade." every 6 months.
"A door is what a dog is perpetually on the wrong side of" - Ogden Nash
I'm confused, I thought that nobody wanted obamacare?
"The WSJ reports that six days into the launch of insurance marketplaces created by the new health-care law, the federal government finally acknowledged that design and software problems have kept customers from applying online for coverage."
What software platform does the software run on ?
I think this problem has less to do with the platform and more to do with the fact that this is what you get when you take the lowest bid without doing some basic research on the competence of the bidder. I mean 92 files per 'Apply'? Seriously? And they rolled it out after the Government Accountability Office warned that insufficient testing had been done? This mess says something about the people running the project. It seems to me that those three months could have been well spent hiring software testing contractors to do some load testing although one gets the feeling from the descriptions that team working on this system were scrambling so madly to get it working by their deadline that there would probably not have been any time to fix any except the very worst the bugs the contractors would have found.
Only to idiots, are orders laws.
-- Henning von Tresckow
Doesn't matter. It's a government job, and everyone involved makes more money if it's a ten-year debacle than if it actually works.
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
Healthcare.gov problems are real. But asking for opinions from people who have a dog in the fight is probably less than ideal. When you ask the likes of Wall Street Journal (Rupert Murdoch's conservative rag) or healthcare technology company EXL (sour that they did not get the contract), you'll get answers that are entirely predictable.
Why is the website a clusterF? Several reasons come to mind.
1. It is a 1.0 product.
2. It is a government project, what do you expect?
3. The states who setup smaller (in comparison) exchanges had similar problems. My state of OR paid Oracle about $50,000,000 for a much simpler setup where you cannot buy anything, but can only view plans on offer. And even that did not work for first few days.
4. The developers were stupid and did not anticipate the traffic they got. Even engineering oriented companies like Google often make that mistake. If you have ever tried registering for Google I/O you would know what I am talking about.
5. Obama's coding skills are simply not up to snuff.
Team Red would like you to think that the govt. has all of a sudden become very inefficient under Obama's presidency. And under their guy Bush, it was a model of transparency and efficiency.
Vermont's site is a disaster. Based on Oracle you'll encounter pages that were set up using what looks like boilerplate language then never corrected. For example, I was prompted to create this one time password – poorly explained – and presented with this screen that tells the user to enter a mobile phone number then shows a field for an email address – there is no field for a phone number. Then, there is a line of text - "I agree to [ENTER COMPANY OR SERVICE NAME HERE] – that is obviously boilerplate that was never replaced or corrected." The pols and the press keep announcing it is a "processing bottleneck" - now blamed on "old computers"... Can you say "we're gonna waste even more money on this thing?"
Oregon did just that. About $50mil later they had a website that did not work for the first few days. And it is a view-only site to begin with.
Giving lots of money to a large company is no guarantee of success.
It doesn't matter if you're behind Akamai if your website is that inefficiently designed. 56 JS files that are downloaded on hitting apply. WTF?
Oracle?
Well played sir. I can't tell if you're trolling or being serious.
Yeah, the communist (not Marxist socialist, but actually "to each according to his need") English NHS is awful.
Oh wait, no, it's the best healthcare system I've ever experienced.
Also the problem here is contracting out to the lowest bidder. The problem was introduction of the private sector into government work - the same problem there always is.
Ofc you're a troll, but a nice launchpad.
Let's examine an HTTP request for a rather beefy portion of the JavaScript in question from healthcare.gov:
They're not even bothering to set the HTTP Cache-Control, Proxy-*, or Expires headers on this content, which will most assuredly limit intermediary proxy and client caching. To say this is amateur hour would be a gross exaggeration of the skills being fielded by these developers.
Much larger issues undoubtedly exist in their backend infrastructure. Given the shit I've seen in this area, I could probably spend the next hour making educated guesses about how badly they've fucked up in various regards, spend another hour partially validating those guesses, and wind up just saying "yup, they're idiots." Instead, I think I'll go to bed now. I have work in the morning.
Write failed: Broken pipe
The waiting time can be a bit of an issue, and a lot of the hospitals are overloaded due to meddling by government officials who have no notion of what it's actually like at ground level, but even through that it still manages to do a very good job of keeping the population alive and healthy. We're beating the US on every health metric worth considering (Except, oddly, cancer survival rate), and at a substantially lower per-capita spending.
Aye, we're not the best on waiting times, and the "internal market" tempered centrally is a lot less efficient than pre-Thatcher, but - like Bevan said - there will be an NHS as long as there are folk left with the faith to fight for it.
Something created out of compassion and solidarity is very hard (and I mean this sincerely) for a more capitalistic society to contemplate, let alone implement.
The USA is frighteningly-close to tumbling into full totalitarianism.
You were doing so well - and then you threw in this bit of unsupported insanity.
Indeed. Remember that Bush/Cheney failed experiment of outsourcing the Iraq War to private companies - companies that brought in untrained "experts" to interrogate prisoners, private security companies to police the streets like the Blackwater employees who killed 17 civilians in Nissor Square, Bahgdad thinking they were being fired upon, or the Halliburton contractor who improperly installed water pumps that killed over a dozen American soldiers while they were showering. Libertarians and anti-government conservatives that complain that government never works while living in a country in which quality of life is almost purely dependent on government programs - like freeways, municipal transportation, clean air, water systems, waste disposal, the internet, police departments, etc, etc, etc - should really just move to Afghanistan.
I'm a bit surprised that we seem to accept the "Obamacare" nomenclature. Can we at least try to be objective? http://www.prosebeforehos.com/video-of-the-day/10/06/obamacare-versus-affordable-care-act/ http://tv.msnbc.com/2013/09/27/poll-more-oppose-obamacare-than-affordable-care-act/
work in progress
That's exactly the quality you get when you outsource to Indian programmers. We've had a decade to evaluate the outsourcing debacle...haven't we learned any lessons from it?
So now Obama can agree to a later start of Obamacare without losing his face: He'll not give in to the Republicans, but just react to deficiencies in the technology.
To add insult to injury, the administration decided to take down the Amber Alerts website, blaming the shutdown, but Michelle Obama's "Let's Move" website is still up. They shut down the PX at Andrews AFB and the WW2 Memorial on the National Mall to WW2 vets, but the golf course at Andrews AFB, which Obama likes, is still open, as is the one at Camp David. Funny what this administration considers "essential".
For this administration it's about not compromising and punishing the American people for supporting their opposition. The pain they intentionally inflict they hope will convince most people to force the opposition to give in. A Park Services Ranger was quoted as saying they were told to make life as painful as possible for people.
"Tell your Senator/Representatives to cave or this kitten (or abducted child that won't show up on the shut-down Amber Alert website) gets it."
1. Nudge
2. Shove
3. Shoot
They are past "Nudge" and are now well into "Shove"...with scattered, mostly kept low-key (for now), but increasingly-numerous incidents where "Shoot" is starting to be employed.
The USA is frighteningly-close to tumbling into full totalitarianism.
Strat
Seriously? You're going to reference The Examiner for the park ranger quote? Come on.
For the rest Reuters has a good explanation of why parts of the government are hit by the shutdown and other parts continue unaffected, the explanation being that the parts that get funding from Congress stop and those and which are funded otherwise continue to function. In the case of the Andrews AFB golf course, for example, it's funded by user fees and is not reliant upon Congress for budget.
Source: http://www.bloomberg.com/news/2013-10-03/troops-forage-for-food-while-golfers-play-on-in-shutdown.html
But hell...don't let details get in the way of your rant...
blindly antisocialist = antisocial
It doesn't matter if you're behind Akamai if your website is that inefficiently designed. 56 JS files that are downloaded on hitting apply. WTF?
When I was young we used a thing called HTML forms.
I guess they don't have enough 'zing' for Obamacare in the 21st century, that's why they weren't considered.
No sig today...
And I'll point out that while WWII started in 1939, but the precepts behind the rise of the Nazis started much earlier.
Totalitarianism does not require mass murder. Especially if the populace is obedient to the authority.
Often times the killer is something stupid like incorrect http headers that prevent caching, which means every request to akamai hits the origin.
Consider Healthcare.gov as an Engineering project. Under .gov procurement rules. . .
The law: an ~1800-page CONOPS document.
The 10K+ pages of accompanying regulations ? User requirements.
So. . .CONOPS passes approval, User reqs start getting gathered. Someone writes an RFP and puts it out for bid. Given typical Fed procurement requirements, that's 9 months to a year before contract award. PPACA passed in March 2010, so we're probably at March 2011 now.
Winner ramps up, develops a Performance Spec and Initial Design, and starts procurement of infrastructure required. Another 6 months. Sept, 2011 now.
Infrastructure stand-up and development begins. Likely another 3 months. It's 2012 now. Standard development and monitoring/audits. Pilot of basic site for Insurance Exchange, though reviews and changes. 6 months min, 9 months likely, Sept 2012.
In the next year, you need to finalize, get the integration between multiple .gov sites and agencies hashed out and tuned, and THEN go to useability, security, and scaling tests. In ANY .gov program, that's 2 years, minimum.
Which means, the first REALISTIC date for Exchange eligibility would have been October 2014. But the lawyers and politicians didn't bother asking the ENGINEERS how long it would take, they never do.
And **THAT**, is my best estimate of what went on and what is going wrong. . .
You, are a fucking moron.
He didn't shut down the ocean.
http://www.politifact.com/florida/statements/2013/oct/07/tweets/did-obama-shut-down-ocean-part-shutdown/
And he didn't shut down the Amber Alert system. The Amber Alert system is a private non-profit entity at the federal level so he couldn't shut it down even if he wanted to.
http://www.politifact.com/truth-o-meter/statements/2013/oct/07/tweets/tweets-and-bloggers-say-obama-used-shutdown-close-/
I don't know how you could ever post something from Breitbart with a straight face.
It has the last-modified header and an Etag. Expires and cache-control are unnecessary. Contrary to popular web developer belief.
http://redbot.org/?descend=True&uri=https://www.healthcare.gov/&req_hdr=Referer%3Ahttps://healthcare.gov/
http://redbot.org/?uri=https://assets.healthcare.gov/global/js/lib/jquery-1.8.2.js&req_hdr=Referer%3Ahttps://healthcare.gov/
HTTP/1.1 200 OK
Server: Apache
ETag: "cfa9051cc0b05eb519f1e16b2a6645d7:1370524513"
Last-Modified: Thu, 23 May 2013 15:59:12 GMT
Accept-Ranges: bytes
Content-Type: application/x-javascript
Vary: Accept-Encoding
Content-Encoding: gzip
Date: Tue, 08 Oct 2013 11:58:37 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Connection: Transfer-Encoding
General
The server's clock is correct.
Content Negotiation
The resource doesn't send Vary consistently.
The ETag doesn't change between negotiated representations.
Content negotiation for gzip compression is supported, saving 64%.
Caching
The resource last changed 137 days 19 hr ago.
This response allows all caches to store it.
This response allows a cache to assign its own freshness lifetime.
Validation
If-Modified-Since conditional requests are supported.
An If-None-Match conditional request returned the full content unchanged.
Partial Content
A ranged request returned partial content, but it was incorrect.
Back when I was still helping with designing and deploying websites, I would always tell clients that they should have a "Simple" backup version of the site. If the problem is load based, there is nothing wrong with having a simple HTML backup system, that generates a way for processing after the transaction is complete. While this might harken back to some of the websites of the late 90s early 2000s, when the CC processor was down, UPS/FedEx/DHL/USPS Shipping Calculation Web Service API or the fulfillment companies XML Order API, it allowed the client to have a sale in hand. It is easier to apologize later and beg forgiveness than to never have the sale. Customer's are amazingly forgiving when you tell them, "We were using our backup system so you weren't inconvenienced, and we have to verify your address, verify your CC info, or the product you ordered is out of stock for several weeks here is an alternative plus something for inconvenience." If they really are pulling from several sources, you trust the user, and when the system returns you run the transactions to verify during normally scheduled low volume times. Also, this is an insurance marketplace, wouldn't your real clients be the insurance companie? Did they not have some say in the testing of the system, or maybe some experience with online ordering systems? Since this is the government, why didn't they do IRS style forms with instruction booklet as a backup. Paper and Pencil backup availability allows them to treat orders like a catalogue order form. I realize all of these backup methods require manpower, but you only have one chance to gain a customer's trust.
Did a nationwide majority vote for a Republican representative, or is the Republican majority in the House purely the result of gerrymandered single-member districts? When you get a chance, search for Redistricting Majority Project, a publicized instance of recent GOP gerrymandering.
Fanatically anti-fanatical
There's a thing called HTTP 1.0, and in it there's a feature called Connection: Keep-Alive. It doesn't spawn a new TCP connection for each of those 56 javascript files. Only one TCP connection per (sub)domain is made when Keep-Alive is in use. This was such a nice feature that in HTTP 1.1, all connections are considered persistent "keep-alive" unless you write Connection: Close. From a network standpoint a few extra lines of HTTP headers between each script isn't going to matter, and if it's cached and/or co-located properly (eg: via Akamai), it actually does matter, since those requests are going to be served from the caches efficiently.
However, the biggest problem is that HTTP is fucking dumb. No, really, it's dumb. Not that it's designers were dumb, just that it's evolved over the years and security was never part of the design. For one, there is no such thing as a "Session". In this day and Age of Information that's ludicrous! Say you use a session cookie to validate every single request for every single resource is valid... because that's what you have to do, then EVERY COOKIE gets sent to the server EVERY TIME you make a request. It's so much face palm, I can feel the back of my skull.
On the security standpoint, neither HTTP or HTML really knows how to actually work with encryption. That happens in TLS. What a fucking crock of shit. HTTPS means you can't cache anything. Most of the files being served are NOT dynamic, but STATIC files. However, since HTTP/HTML are so fucking dumb they can't even provide a simple hash, then you can't trust mixed content. If in addition to the URL of a static resource, you could also include a known hash:
<img src="..." digest="d8b09c45b522e34d81ac9eed95f922c7028e7fb2; type=hex/SHA-1">
Then the browser could hash the unsecured (cache-able) resource as it's pulling it in at the behest of the secured dynamic (uncatchable) page, and verify that the requested unsecured content wasn't tampered with in transit so it wouldn't be a security issue and we could actually FUCKING USE SECURITY EFFICIENTLY, grrr. Especially if you could specify a few bits of salt with the hashes...
<img src="..." hmac="WkRoaU1EbGpORFZpTlRJeVpUTQo=, TlRJeVpUTTBaRGd4WVdNNVpRbwo=; type=base64/SHA-1">
But, no, that doesn't exist. No HTTPS content is cached. Apparently I'm the only one on the planet not drinking the damn cool-aide. The web is bloated and retarded, it needs to die. Long live the Internet, but fuck the web. It took HALF the age of the Internet just to get from HTTP 4.01 to HTML 5... Over a Decade, and this shit still isn't in the spec. Don't hold your damn breath for next version, or for anyone with a fucking clue how things should work to propose sane changes. Even Google with SPDY is just exacerbating the issue with bandaids over the inefficiencies of HTTP.
TL;DR: Yeah, it's a shitty website / backend design, but primarily it's because HTTP/HTML is just fucking retarded.
The ACA is the compromise. The Democrat idea was single payer Medicare for all. We compromised and used the Republican model proposed in the 90s, and implemented a decade later by Republican Governor Romney in Massachusetts. The ACA passed both houses of Congress. It was signed by the President. It was upheld by the Supreme Court. The Republicans in the house tried over 40 times to repeal it and failed. They are now throwing a temper tantrum because they can't get their way through the established, official, channels. They're not willing to accept that they lost this fight. They'd rather burn everything down than see the other side score a legitimate victory. It's scorched Earth. Spite.
If you build it, nerds will come. Soylentnews.org
This is just one of those things that the government really doesn't do all that well. Private organizations live and die by their profit margin, so they make damn sure shit works and it works affordably.
I cannot let this comment pass. Sorry, but anyone who's worked for a large corporate beauracracy knows this is nonsense. They are just as large, Byzantine, and wasteful. That's simply how large human organizations function.
Natural != (nontoxic || beneficial)
When you don't use the word "Obamacare" and you go through the ACA provision by provision, it's overwhelmingly supported. You have to use scaremongering and knee-jerk words, to get people to say they are against it. Ask people, do they think insurance providers should be able to deny coverage based on pre-existing conditions? Overwhelming answer is no. ACA does this. Children stay on until 26? They answer yes. ACA does this. And on and on.
And why do you think the Democrats controlled all three branches? How did that happen? They were voted in.
The Republicans biggest fear right now is that they won't be able to stop the ACA in time before people start seeing the benefits, and then they'll never be able to get rid of it just like Social Security and Medicare. Once people see first hand that social programs can actually work, and work well, it becomes a lot harder to sell their private market, anti-government, rhetoric. The ACA is a threat to their brand.
If you build it, nerds will come. Soylentnews.org