Tech Titans Oracle, Red Hat and Google To Help Fix Healthcare.gov

wjcofkc writes "The United States Government has officially called in the calvary over the problems with Healthcare.gov. Tech titans Oracle, Red Hat and Google have been tapped to join the effort to fix the website that went live a month ago, only to quickly roll over and die. While a tech surge of engineers to fix such a complex problem is arguably not the greatest idea, if you're going to do so, you might as well bring in the big guns. The question is: can they make the end of November deadline?"

Comments

Answer: No.

[Anonymous Coward]

Nine women cannot make a baby in one month.

Re:Answer: No.

[BasilBrush]

Nine women cannot make a baby in one month.

True, but the website already exists. If it's a case of fixing defects rather than re-architecting from scratch, there's no reason why multiple teams can't work on different parts of the system. And multiple people within a team can't work on different defects.

Defect fixing is indeed somewhat scalable.

Re:Answer: No.

[Virtucon]

No but I heard 18 Women can do it in two weeks. The guy from Infosys told me so.

Re:Answer: No.

[turkeydance]

the Mythical Man Month returns

Re:Answer: No.

[dmbasso]

It all depends on the quality of the existing code base. More often than not, it's better to start from scratch.

Re:Answer: No.

[MightyMartian]

The last time I had to "re-architect" an existing website, I ended up putting in roughly twice the amount of time as the original "architects" (and I use that word very very loosely). Believe me, there's a lot of shit out there that will require a lot more effort to fix than originally went into building it.

Re:Answer: No.

[sexconker]

there's no reason why multiple teams can't work on different parts of the system

You've never worked on anything with multiple teams working on different parts, have you?
It never fucking works. You need knowledgeable oversight.

Re:Answer: No.

[icebike]

Nine women cannot make a baby in one month.

But I bet even one woman could spell cavalry, and know the difference.

Slashdot editors wanted. No Experience needed. We wouldn't know what to do with experience if we tripped over it.

Re:Answer: No.

[MightyMartian]

Exactly. I think the Linux kernel is a damned good example of how a large number of developers working in very different kinds of development environments, some working in side-projects like Netfilter, are coordinated by one guy intimately acquainted with the kernel.

You can say what you like about Linus's attitude at times, but the fact that the Linux kernel is running on everything from supercomputers to be Nexus 7 tablet tells you that there is a way to successfully and productively organize multiple teams to produce a successful software product.

Re:Answer: No.

[Jstlook]

My wife saw that book on my shelf last night and asked if it was related to a man's period. I had to chuckle.

Re:Answer: No.

[Anonymous Coward]

That article is so full of contradictory statements it's ridiculous. Which isn't to say I'm defending the excessively sugar-coated defenses the administration made in 2010. But 94 million is an upper limit, and it's mostly composed of private insurers and private companies purposefully choosing to change coverage, not because the law mandates it.

And let's not forget about the 20-40 million people who will be unable to keep their lack of insurance coverage. What's the difference between being uninsured and underinsured? Maybe I should be allowed to get a car insurance policy with a $100 limit. I mean, freedom, right?

If you want to diss the ACA, then diss it on its merits.

I hate taxes as much as the next guy. More, in fact. My combined income is over $240k/year, almost all earned income, so its taxed heavily. It's a gigantic bitch. But you know what? I grew up in poverty, in foster homes. I benefited from a safety net. And the elder members of my family all depend on some sort of government assistance. So I just suck it up, because as the extremely conservative Justice Holmes once said, taxes are the price of civilization. And this civilization let's me make almost a quarter of a million per year. You think I could make that in Mexico, Brazil, or China?

The penalties for having no insurance are is like $150/year. If you can't afford that, then you have bigger problems--and in any event, if you couldn't afford it the government would pay for it.

Re:Answer: No.

[erroneus]

Interestingly said.

Even if by some miracle, they bring something up, it doesn't fix the actual problems. Ridiculously increased rates, it's a new tax on everyone, lies about keeping one's old policy and a general over-all burdon on the remaining who are employed above the poverty line.

Re:Answer: No.

[kimvette]

> True, but the website already exists. If it's a case of fixing defects rather than re-architecting from scratch, there's no reason why multiple teams can't work on different parts of the system. And multiple people within a team can't work on different defects.

You are assuming that there is a detailed (and accurate) functional spec, design spec, and that the code is organized and well-documented - and that it is architected in such a manner that throwing more engineers at it will actually fix the problem. More often than not, that is not the case.

Re:Answer: No.

[Fluffeh]

They didn't mean like horses and stuff... Wow, they like totally meant Calvary - cause that's like the most common saying ever, you know, calling in the ancient name for Golgotha, the place just outside Jerusalem. You and your horses.... cavalry indeed. Preposterous!

*sips coffee*

Re:Answer: No.

[craigminah]

...and let's hope the US Government has enough sense to not pay the clowns who built that FUBARed website. Interesting Michelle Obama’s Princeton classmate is executive at company that built Obamacare website and won the contract in a rare no-bid contract. Very fishy...

Re:Answer: No.

[Snotnose]

But only if they're Indian or Chinese women, American women are too lazy.

Re:Answer: No.

[funwithBSD]

Someone is going to be crucified before this is over...

Re:Answer: No.

[DigiShaman]

Being that
1: this code was already created by inexperienced developers.
2: anything created from the group of these titans will by default be superior work compared to the last guys.

Re:Answer: No.

[recharged95]

But all 3 companies listed will have those rock stars that will:
a. look at the code and call is rubbish.
b. ask to rewrite the whole thing
c. charge an arm and a leg to do it within time.
d. run it under agile (so THEY control the requirements, not the domain experts).

Really they should have hired the guys that do turbotax and such.... it works for the type of users on this healthcare system. The above 3 will struggle through it as well... but will milk it for all it's worth.

All I say to the Obamacare management team & Obama: TAKE A STEP BACK, WAIT.... ASSESS THE PROBLEMS one by one, THEN HIRE THE RIGHT FOLKS. This is a knee jerk reaction and will go down in flames. Of course, the valley and wall street is loving it....

Young MBA folks: this is your Y2K computer problem moment. Remember those times: the panic, the flooding of cash, and nothing happened afterall? Yeah, get ready for another internet boom/bust.

Re:Answer: No.

[Anonymous Coward]

Rare? Nearly every government contract offered to the private sector since Bush took office has been no-bid. Remember the deals made during the Iraq War? Every single one of those was no-bid to Halliburton. This kind of cronyism is NOT rare at all; it is the norm, and has been for over a decade.

Re:Answer: No.

[Beeftopia]

Almost universally in software development, starting from scratch is a stupid fucking idea repeated by inexperienced developers.

For working, debugged, stable code that looks messy, that's almost always true. But this site failed immediately when put under load.

Now, if the site logic for ONE user is sound, then they could preserve that and put in the infrastructure needed to handle the sheer electricity of thousands of requests per minute. That's what Google is known for, with their vast datacenters and ability to load balance. Oracle is known for databases able to handle high concurrent transaction loads. Red Hat can provide support on a reliable, robust operating system (Linux).

For 654 million dollars, hopefully the government got the logic and blueprints down for how one user is supposed to progress. Now, the folks who know how to handle the sheer electrical volume of the massive numbers of connections can perhaps install that missing, essential portion of the website. IF of course, the design and logic of the site for one user is sound.

Re:Answer: No.

[Fluffeh]

But be back with more wiseass comments in three days...

Re:Answer: No.

[mysidia]

Almost universally in software development, starting from scratch is a stupid fucking idea repeated by inexperienced developers.

When the code is an unsalvageable pile of crap; sometimes it does make more sense, to reevaluate the design, and re-implement the entire application properly, using the old code only as a reference; than to try and repair.

Vermont's Site is Toast

[Cornwallis]

Our Gov is finally "out of patience" with Vermont's site (built by the same CGI that did such a bang up job on the Fed system: http://www.burlingtonfreepress.com/article/20131031/NEWS03/310310034/Governor-Peter-Shumlin-Web-woes-prompt-changes-to-Vermont-health-reform

Re:Vermont's Site is Toast

[BradMajors]

All Vermont needs to do is buy a copy of Kentucky's system. Kentucky's system works fine.

Amazon

[qzzpjs]

I think they should have just listed the plans on Amazon. Almost everyone already knows how to buy stuff from them and their servers would have handled it.

Re:Amazon

[Ralph Wiggam]

Prices and availability vary hugely for the same insurance plan for different people. Amazon has no way of handling that.

Re:Amazon

[Xyrus]

Just what we need. One-click insurance from Amazon. :P

Calvary?

[themushroom]

It's a Biblical reference -- and at this rate it would take divine intervention.

Re:Calvary?

[icebike]

It's a Biblical reference -- and at this rate it would take divine intervention.

This is government, nobody gets crucified, they all get promoted.

Re:Why not IBM

[BasilBrush]

IBM certainly made sure the Nazi's CRM system worked right.

Let's see..

[Virtucon]

In two months the site will be using Oracle and Ellison will charge the Feds a fortune for the license fees.
Google will start mining every piece of data it can get off the website, of course the NSA will be stealing that and stashing it in Utah.

Red Hat will push it all to RHEL which isn't necessarily a bad thing.

Red Hat?

[Anonymous Coward]

No Microsoft? lol :)

Re:Why not IBM

[Virtucon]

No, they wanted it done and not outsourced to India.

Oracle? Seriously?

[jcr]

I guess nobody in the decision making loop heard about Oracle's big California DMV fuck-up.

-jcr

Brooks Law

[mccrew]

Brooks Law states "adding manpower to a late software project makes it later".

Re:Oracle! YES!!

[jbengt]

I've had the misfortune of needing to use an Oracle system with a web interface to deal with a large client for construction management & billing. If that experience is any indication of how Oracle will fix the problem, the Feds would be better off keeping the very crappy existing system. (seriously)

Re:Just say no

[zippthorne]

In that scenario, we'd actually be worse off - the ones with principles wouldn't be working on it...

Re:Calvary? Really?

[cold fjord]

Of course maybe it was a literary illusion. ;D

huh?

[Charliemopps]

I can understand Google and Redhat... but Oracle? Talk about having a fox in the hen-house.

Too Many Cooks Spoil the Soup

[liwee]

Enlisting JUST ONE of the tech giants would be more productive.

Re:Will they teach Economics?

[Skapare]

The government should have done it in-house, using directly hired citizens as developers and project managers. Use top developers that fully understand the selected technology. This site is something that will be changing a lot over many years, so continued staff where most developers already know how it's built would keep it upgraded.

Google?....

[tooyoung]

Crap, now the NSA will have a backdoor into the government!

Why can't they start over ?

[Taco Cowboy]

Instead of fixing a bunch of hopeless code, why can't they start over the damn thing - with a properly designed paradigm ?

Re:Why can't they start over ?

[Runaway1956]

That's my idea. Government hired an incompetent contractor to build something. They built a freaking MESS. Just clear it all out. Sure, examine the code, see what the ideas were when they built the site. Take the best ideas, and rebuild the ideas, from the ground up.

Years ago, I was called in to a construction job, where the previous foreman had really screwed up. He built a foundation and wall in the wrong place. We didn't try to make the wall fit into the plan - we wrecked the frigging wall, poured a new footer, and built the wall on top of our new footer.

The site designers need to do the equivalent. Consider the "blueprint", see where everything went wrong, tear out the screwups, and build from the ground up. If that should happen to mean that not one single line of code remains, then so be it. If it means that 1/4 or 1/2 or even 3/4 of the code can be reused - fine. Just get it working. And, do it for less than another half billion freaking dollars!!

Re:Why can't they start over ?

[game kid]

And, do it for less than another half billion freaking dollars!!

Oracle's involved, so good luck with that.

Re:Why can't they start over ?

[gweihir]

Actually, government hired far too many contractors as everybody wanted a piece. Now they are doing the same again. Have one competent entity fix this mess, not a lot of them and especially not a lot of them that are not used to cooperating.

Re:Why can't they start over ?

[gtall]

HHS was supposed to provide the supervisory role. Problem was they didn't have the experience to do such a thing. In a way, they were stuck. If they'd've hired a single contractor, they'd still be in litigation because the others would have sued. Hiring many meant they couldn't use a single company to ride shotgun because companies don't play well together in shotgun marriages.

They should have had the NSA do it. I hear they are quite good a building large systems.

Called in the calvary?

[goosebane]

Alright, who is getting crucified over this one?

Re:And...

[the eric conspiracy]

It's not 600 million.

http://www.washingtonpost.com/blogs/fact-checker/wp/2013/10/24/how-much-did-healthcare-gov-cost/?wprss=rss_politics

Re:Oracle! YES!!

[FlyingGuy]

One of the many problems is that most people do not know how to tune Oracle. Properly tuned Oracle, even when running on inadequate hardware, oracle can support TPS levels that many DB's only dream about with full ACID as a matter of course on the same hardware. I have watched Postgres, MS-SQL Server and DB2 just hit the floor while Oracle kept chugging right along, not always mind you, but more often then not.

I am currently running 11gR2 on hardware that is at best adequate and can assimilate the entire output of 80% of the state of California's highway loop detectors ( approximately 50,000 raw data rows inserted every 30 seconds 24/7/365 ) and that into a rather poky 15TB drive array with 7500rpm 2TB drives, in raid 5 no less, then query all of that data filter,clean and analyze it and shove that data into another table all in the same 30 second period.

The DMV project was a nightmare of never ending changes of requirements. When you think about the basic project, it aint that hard, but when there is no point at which you could say it was stable because the target just kept moving, I don't care who takes it on or who's DB engine you throw at it, it will fail.

When it comes to scaling something out, you take you best guess at what you load will be. When your prospective load might be a large percentage of 300 million people it is a hard target to pin down and that is what ( along with a few bugs that escaped unit testing ) was their ultimate undoing. No one knows who's DB engine was behind it but I doubt it was any of the "web scale" DB's since they don't support ACID very well and this was one of those when it was absolutely essential.