Slashdot Mirror
NYT: Healthcare.gov Project Chaos Due Partly To Unorthodox Database Choice
First time accepted submitter conoviator writes "The NY Times has just published a piece providing more background on the healthcare.gov software project. One interesting aspect: 'Another sore point was the Medicare agency's decision to use database software, from a company called MarkLogic, that managed the data differently from systems by companies like IBM, Microsoft and Oracle. CGI officials argued that it would slow work because it was too unfamiliar. Government officials disagreed, and its configuration remains a serious problem.'" The story does not say that MarkLogic's software is bad in itself, only that the choice meant increased complexity on the project.
Who owns this company?
how much do they contribute to XXX???
There has got to be some reason that this DB that ive never even heard of (and i work with DBs, its not my main point of work but I know my way around DBs) got the gig over the more established players.
or, perhaps they went with it because it is less known and therefore reduce the risk of known attacks in other DB systems?
have you seen my sig? there are many others like it but none that are the same
FTA: "An initial assessment identified more than 600 hardware and software defects — 'the longest list anybody had ever seen,' one person involved with the project said. "
Strikes me as none of these people seemed to have ever worked on large projects before.
You could take a handful of proven DB technologies such as Oracle/DB/MSSQL, throw a web (Apache/IIS) and app (.Net/WAS/Jboss) front end to it, and it would work. Why did these guys fuck up the whole thing? It's like the scene in The Fountainhead when the second-rate architects smash up the plans and add their own stuff, "to express their own individuality". This could have been a solved problem - hell, it WAS a solved problem.
I want to delete my account but Slashdot doesn't allow it.
A little googling turns up that MarkLogic's offering is NoSQL. Without getting into the whole SQL/NoSQL debate, I can't help but noting that this is clearly relational data with a fairly limited number of records (clearly there can't be more than 300M people listed) and for which ACID is (or should be) a major concern.
And here is a fantastic example of what happens when hype trumps common sense. NoSQL is the new hawtness, and apparently the dumbasses running the project wanted to be part of that. Now MarkLogic, and NoSQL in general, will have a massive blow to their reputations, and it's unknown how badly this will hurt them.
As someone who has done databases for a long time, I have very little respect for NoSQL, but that is mostly because everyone keeps trumpeting it as a killer of traditional databases. There are scenarios where NoSQL systems are an ideal fit. However, NONE of those scenarios require data to be very reliably stored in a guaranteed and predictable way.
If you don't get your tweets or your friends facebook posts as soon as they are posted, no one will really care. But for something as truly important as health insurance coverage? Are you f__king kidding me? And that's just from a reliability standpoint. Nevermind the fact that NoSQL is currently at the wild west stage where nobody is compatible with anybody else, there is nothing resembling a standard set of APIs between products, making it very difficult to develop expertise.
Sounds like the Gov was just begging for problems.
"Some people, when confronted with a problem, think 'I know, I'll use XML.' Now they have two problems."
-JWZ
MarkLogic is an XML database, not a relational database, so if your data primarily consists of XML content then it's the right tool for the job. Sounds like the vendor building the system had a favorite hammer and decided that a rather traditional database problem looked like a nail.
MarkLogic itself is fine if your data fits neatly into an XML schema, but with healthcare.gov that tree is probably enormous and hard to optimize for DB activity.
Software Shouldn't Suck
E-mail: frank at jacquette dot spamless com (remove the spamless!)
It was designed from the start to pave the way for single payer health care.
Bullshit. Since the law has been passed almost the entire implementation has been up to the administration. Which in case you had not noticed is lead by partisans in favor of the affordable care act.
If the GOP can be accused of anything material in terms of interfering with the implementation it would preventing the law from being amended, and if it needs amending there is your proof it was bad law to begin with. Sorry Obama owns this and really HE owns it not even the larger DNC as its HIS branch of government that has been responsible for implementation from the moment HE signed the thing.
*IF* it fails its either because it was unworkable in the first place as written and should not have been enacted or because the Obama administration failed to execute. Those are really the only honest high level conclusions.
The argument well because they had to pass the Senate bill as is it did not get the usual fixes and tweaking is why its got so many problems is also bogus. They had to pass the Senate bill by abusing the budget reconciliation process to deny the minority party in the House at the time its rights; they knew if usual procedures were followed it would never have become law. So again it either should not have been enacted or the administration has failed to execute.
Repeal the 17th Amendment TODAY! Also Please Read http://www.gnu.org/philosophy/right-to-read.html
Well they did have the flashiest ads which used lots of buzzwords.
Undetectable Steganography? Yep, there's an app fo
Why not just use MySQL, MongoDB or MariaDB? At least use a database system that has good support, an easy learning curve and loads of followers. That beings said, proper testing would easily of mitigated this entire issue.
Since the law has been passed almost the entire implementation has been up to the administration.
The implementation is, for the most part up to the administration and they certainly have screwed the pooch on this; however, the GOP can quite easily be accused of willfully sabotaging the process through the ACA lawsuits and intentional state exchange shortcomings (not to mention a myriad of other issues that relate to the shutdown, refusing ancillary funding out of spite, and issues with the Supreme Court.)
In any case, even if the GOP are being total jerks about the law, healthcare.gov requires a software solution that has been patterned thousands of times. It's a total fumble by the Democrats.
The party system should simply be banned outright.
Loading...
To be fair, the AC didn't specify Republicans and Democrats. It could have been internal groups behind the website itself that wanted to blame other internal groups, to pump up their own importance. That could be what he meant.
.
But, yeah, he almost certainly made a laughable dig at the GOP there.
If you think I voted for Trump because of this post, you're wrong. I voted for Dr. Jill Stein of the Green Party. Again.
That's ok, your share of taxes actually was specifically allocated to paying for a toilet in the Ozarks.
Just when I think.. wow.. you couldnt screw this up any worse... they step up and prove me wrong.
The "47%" that people keep talking about are almost entirely retirees, adults going to full time college, or the disabled. The largest portion of the group are the retired people. Are you saying that people who have worked their whole like and are now retired, should not have a say in taxes?
I've lost count of how many projects I've been on where the architect decides to "make his mark" by using unconventional design choices. Then the project gets stuck in a dev hell where the actual developers struggle with either integration headaches or difficulties with the code not acting like they expect. There's something to be said for plain vanilla.
Everyone pays taxes, income or not. Buy something? Sales tax. Pay utility bills? Taxed. Buy gas? Taxed.
SJWs are the new boogeyman. -Me
Also the working poor - 47% pay no _income_ tax, but more than half of those people do pay payroll taxes, and even many those who don't (after deductions) still work.
They also pay sales tax, property tax, etc etc.
I worked as a contractor developing a system at FDA. It lasted for 5 years. Inside the Beltway, it's pretty much the same all over. Dysfunctional communications and ridiculous breakdown of authority not corresponding to lines of management. No accountability. Project management requirements that have never been followed by any project. No commitment to the output of requirements gathering. No requirements change control. No performance engineering. Inadequate testing. No acceptance process by the government. IT groups with oversight for contractor output that have never written a line of code. All in all, pretty sick and ugly. Prior to my project there were 5 failed attempts. My project followed PMI practices, worked them hard and succeeded.
See: http://assets1.csc.com/innovation/downloads/LEFBriefing_MarkLogic_031512.pdf (slide 23)
My two cents:
GOP can quite easily be accused of willfully sabotaging the process through the ACA lawsuits
Attempting to sabotage it maybe, succeeding no. There was never even so much as a stay issued, as far as I am aware, preventing the administration from getting started on implementation. If I am wrong on that please correct me but I don't think the court cases could have had much impact on the implementation.
The medicare expansion issue is going to create a coverage gap, in the states that did not do it. That is about the only *problem* the legal challenges successfully created.
As far as states not implementing their own exchanges the law always intended to provide a federal solution for them; so that isn't much of an excuse for its failure.
Repeal the 17th Amendment TODAY! Also Please Read http://www.gnu.org/philosophy/right-to-read.html
Sorry to break it to you but that "statistic" is a bunch of BS. Everyone in the US pays taxes of some sort. I'm not particularly high on the tax scale (about $36k a year) and I estimate that i pay at least 30% of my income in local, state, federal and specific taxes (sales, gas, property, vehicle, etc). MAYBE some very poor (or very rich) people can avoid some of those taxes but EVERYONE pays at least a few of them.
From the summary:
"The story does not say that MarkLogic's software is bad in itself, only that the choice meant increased complexity on the project."
Unless that complexity was necessary to solve a problem, then it is in fact bad.
Please use MY tax for that - its way better than most of the stuff that tax gets used for! (On condition that it is a unisex toilet).
Sent from my ASR33 using ASCII
We were looking at what we could figure out about the architecture of healthcare.gov, and one problem is that it looks like it's using Oracle Identity Manager to manage the permissions of what users can/can't see. That means that OIM is burned in - and it's probably brutally slow, since every time you need to check a permission you go through OIM.
I'm not positive that's the case, but it fits given what pieces of the architecture I've seen. It would also explain why the system doesn't perform - permissions checking is always brutal, especially if you don't cache them. Caching permissions has more issues.
Bad news -- it's a glory hole. Still a better use than most tax dollars.
Do you even lift?
These aren't the 'roids you're looking for.
It is NEVER the tool, but who and why it was chosen.
Furthermore, MarkLogic is a legitimate NoSQL vendor with strong ACID.
So the question is "how does MarLogic screw up the site", without the answer to that question, we should all refrain from pointing finger to merely a small piece in a huge software project.
NoSQL is NOT the new hotness, it's been out there for at least 5 years and many successful projects are using them, so for the ones that things NOSQL shouldn't be used, wake up and breath some fresh air.
It's even more basic than that. According to TFA the design goal was: "creating a cutting-edge website that would use the latest technologies to dazzle consumers with its many features. "
In other words, even if they had gotten this thing up and working as they wanted, right on time, it would have been an accessibility nightmare that would never work unless you have a brand new computer configured with no security anyway.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Friends don't let friends enable ecmascript.
Uh, no the Republicans do not have clean hands in this foaled rollout.
By refusing to set up the state exchanges they shifted a massive burden to the Feds unplanned for in the law. Then, they refused to expand funding for this massive integration effort. Think about it... Most of the state exchanges are doing fine. Why? Because they only have to meet one state's requirements. The Feds were forced by Republican tactics to handle the differing requirements for 30+ states.
"The story does not say that MarkLogic's software is bad in itself, only that the choice meant increased complexity on the project. "
But the subliminal take away is that MarkLogic was used and things went poorly. I feel for the people at MarkLogic in that they may succumb to someone else's stupidity even though their product is fine.
So why that decision? Whose family member is in that company?
"If any question why we died, Tell them because our fathers lied."
So the private-owned company has "officials". Cool...
none
But how would a traditional relational database scale to the 1 billion, or 1,000 billion users, huh? Did you think about the need to future-proof the application?
Current population of the U.S. is a little over 300 million. That includes children, people who have company provided health insurance, etc. who don't need to access HealthCare.gov so that the number of users of HealthCare.gov is expected to be about 30 to 45 million people.
The only way HealthCare.gov would need to support 1 billion or more users would be if we inflicted it on the China or India. That could lead to war or poor technical support and customer service. Although the later does raise an intereting question of who does someone at a call center call when they need technical support? Likewise, the population of the planet is between 7 and 8 billion. For the 1,000 billion users are you planning on also providing health insurance to E.T?
Cheers,
Dave
They that can give up essential liberty to obtain a little temporary safety deserve neither safety nor liberty.
Ben
I get there must be tons of complexity in managing healthcare.gov site interacting with all necessary stakeholders... Must be quite a lot of different databases, systems, operators to say nothing of complexity of working the actual problem space. I can understand how there might be glitches that cause wrong rates and plans to be communicated.
What I am still puzzled by are the "waiting rooms" with 40k people waiting to use a site. What the hell can justify it being so computationally expensive to spit out a list of plans? Does the universe need to be recalculated every time someone signs up? Is there some manner of massive graph problem needing to be solved for each user? If you ran a profiler on the web site what would it most be spending its time doing?
It uses NoSQL. Which does add a serious amount of complexity to the picture. SQL by comparison, row based, column addressable and relational. Gee, what could be so wrong with that?
The GOP is more Machiavellian then you give them credit for. All it takes to cause interference is to buy off some insiders that will make strategic blunders that are beneficial to ones goal. Malice masquerading as incompetence is the safest way to stop up the works in DC. As long as the money trail can be hidden there is no risk of blowback.
I am becoming gerund, destroyer of verbs.
The system had to coordinate with multiple independently run systems from multiple agencies and wasn't led by anyone with authority to dictate terms or experience in ugly systems integration.
As a European living in a country with a fairly well working public health system (Sweden), and a tech background, I am amazed that Slashdot and other tech media is behind the curve on this story. Why is NYTimes and other mainstream media the ones running with this and finding the background etc? Damn common sense tech people in the U.S.! Get on top of this to fix it!
Die dulci fruere. Have a nice day.
The real problem with this website is too much ambition. They should have started with the simplest working project, something like Health Sherpa, proved that it worked, and then tried scaling up from there. That's the Lizze Borden school of web design. Cut your design down to the bone. Just when you think you're done, no matter -- there's always one more whack you can make!
I felt bad for the man who had no signature, until I met a man who had no comment.
The problem is this:
Government officials like issuing "cost plus" contracts because they allow them to remain in control of the work while not doing any of it themselves. They probably wanted to use it because they had used it in other projects in the past. If the'd just issued a fixed pice contract and said "get it done" the contractor would have selected a database they know and everything would have been much easier for them.
My car insurance doesn't cover oil changes; my homeowner's insurance isn't going to cover replacing a screen door that went kaput during the storms last weekend; so why the hell does health insurance cover routine check-up visits?
Because if it doesn't, then a lot of financially strapped people will skip those visits, and incipient problems won't show up until later, when they are much more expensive to fix. Funding preventive care actually saves money in the long run.
MUMPS is used by the Veteran's Administration, and probably others too. In fact, it became an ISO/IEC standard in 2005.
Intersystems Caché and FIS GT.M are the most popular implementations. GT.M is open source and available under the GPLv2.
http://en.wikipedia.org/wiki/MUMPS
Since the days of computers first came into common use there must have been tens of thousands of database programs written. There are probably thousands of database projects underway as you read these words. And somewhere today, at least one of them is failing. ... Databases are wheels that have been re-invented so often that many veteran developers could stumble through such projects with their eyes closed. Yet these efforts sometimes still manage to fail.
Attempting to sabotage it maybe, succeeding no. There was never even so much as a stay issued, as far as I am aware, preventing the administration from getting started on implementation
Sorry, but even as an independent I would have to disagree with that. The statute provided monies for states to create their exchanges, there was no stipulated money to fund a federal exchange because it was entirely expected that states would want to control this themselves.
Suddenly GOP led states started saying "we're not building that" which makes absolutely zero sense except as a political stratagem to sabotage the process. You should note that the only two non-GOP states to leave the burden entirely to the feds are Montana and Missouri. Montana wasn't able to build their own because the GOP controlled legislature killed the governor's attempt to begin designing the exchange. Missouri is doing this for EXACTLY the same reasons.
The ambiguity over how many states the federal system would have to account for apparently hampered the setting of specifications (it would seem that most states needed custom integration bridges into the federal system) and severely impact the costs and complexity of the project (which was funded through means other than the bill.) It was always the goal of the statute that as many states as possible would manage their own exchanges - it was clearly not expected that the GOP would directly urge Republican led states to add their load the the federal exchange out of spite for the ACA. The irony is that this is the GOP asking states to give up authority to the federal government, lol.
You'll note that the states that did build their exchanges with the federal monies set aside for them seem to be doing ok, many of them exceeding their target roll-out goals.
It's also kind of silly to argue that GOP opposition hasn't made it more difficult and complicated to build, roll out, and run Healthcare.gov, I mean, simply ask yourself - "Would the healthcare.gov rollout gone better or worse with support from the GOP?"
The GOP is clearly doing everything in its power to have the federal exchange fail, and "doing everything in their power" doesn't mean just sitting quietly as you seem to suggest, it is actively sabotaging the funding and the process.
As for Democrats, apparently they ...couldn't build anything other than a good case of hemorrhoids.
BTW, totally agree with you regarding the 17th amendment - states lost virtually all of their ability to be effectively represented when this happened.
Loading...
MarkLogic is a good product when used as intended. Bad use case here. Like driving in wood screws with a hammer - defeats the intended purpose of the screws and causes things to fall apart.
Since the law has been passed almost the entire implementation has been up to the administration.
I think you meant:
Since the law has been passed almost the entire implementation has been up to the bureaucrats.
FTFY.
VAT is disproportionally more painful for those without means than those with a lot of money.
I don't approve of my taxes being used to wage war on people we have no right waging war on. Guess what - don't have a say in that either.
I've used it, personally, to implement a public-facing website. That site endured the dreaded 'slashdot effect' several times. No failures.
When implemented properly, Marklogic is damn near unkillable. It will slow down, it will reject connections when queues are full, but it will not fall over. Naturally, this assumes proper underpinnings and capacity calculations. With Marklogic, those are actually documented.
Mandatory disclosure: I do not have and have never had any association with Marklogic other than a paying customer.
I don't know about YOUR health care insurance, but mine decided that if Medicare didn't consider a routine physical to be worth covering, then they wouldn't cover it either.
I think we've pushed this "anyone can grow up to be president" thing too far.
A few weeks ago, the press did a bunch of reporting that the Gov had reached out to Google and Oracle to come help fix their website. Haven't heard anything else since. Anybody know what happened with that move?
Given what was in this article, I wouldn't be surprised if the Google engineers showed up, looked at the mess and said, "yeeah. good luck with that." and walked out. Maybe even said, "We'll help you if we get to throw out everything and rewrite it clean in a couple of months and host it on the Google-plex."
That's kind of the point and one of the reasons, perhaps, that the free market or even the insurance model doesn't work for health care. I also read an argument once that the reason that chiropracty and homeopathy work is because the provider spends more time with the patient and thus obtains a better understanding of the risks that patient might be subject to. Of course, homeopathy itself is bunk; and chiropracty might have specific uses, but I lol every time I see the banner down the main drag in town that this chiropractor can cure ear infections. However, it does seem to reason that better outcomes would result if patients and their providers shared more information. Profits create a perverse incentive that minimize the contact between patient and provider.
Join the Slashcott! Stay away entirely Feb 10 thru Feb 17! Close all tabs to prevent autorefresh!
NoSQL and Notes types databases are ideal when volume is low, and variety is high and needed values change frequently. Seems to me the data they were collecting and presenting are ideal SQL candidates, and terrible NoSQL candidates. Even if the DB is not bad, it IS a very bad choice for this type of application. Although I'm sure the salesperson/campaign contributor would disagree.
I'm sure it is more difficult to 'hack' simply because it is less familiar to hackers, but validated input, and well patched servers is a better way to prevent getting hacked.
(If at first you don't succeed, do it different next time!)
I used MarkLogic at a customer for over 8 years and its the best technology we used by far. This is a blame-fest and I never had any problems with the technology. MarkLogic scales really well and our website had over 300 hits per second AVERAGE to it daily and MarkLogic was never slow and always performed. In something like this I would look towards the group doing the finger pointing, thats usually a good place to start!
That's funny, I didn't approve of my taxes being spent killing hundreds of thousands of people that were no threat to me or anyone I know, yet it still happened. WTF?
Starting in 2009, the Government's data strategy has emphasized data interoperability across organizations. MarkLogic has a fine product that optimizes XML communications between and among organizations. Because the public sites' data stores would be transmitted among the states' repositories, it's easy to see that data strategy would center on exchanges in the chosen lingua franca. Contrary to many of the posts here, I will not take issue with XML. Using it is complex, and it requires verbose communications, certainly - but the real issue with the healthcare.gov fiasco was too many moving parts. I have never seen a fully functional XML capture-and-exchange DB, but I have seen several long, drawn-out projects that sought to produce such a DB as their end product. For a mission-critical system such as healthcare.gov, they should have built two prototypes, one with XML architecture as the to-be model, and the other with a functional architecture that could evolve into an XML structure - but engineered for risk reduction. The real problem here was ineffective risk management, and the ship just kept running toward the rocks while the helmsman held the wheel steady.
I should make clear that I'm posting on the basis of personal experience. I have no particular ties to MarkLogic (though companies I've worked for do use the technology), or the US government.
Firstly, the NYT article is poorly researched. My sources tell me that MarkLogic Server is far from the only data storage technology and vendor involved, and that while the MarkLogic-powered aspects of the system did require some remedial action, those aspects were not central to the publicised problems. Remember when MS used their press contacts and marketing clout to smear OSS on the server back in 1999? I wouldn't be surprised if that's going on here.
Secondly, I have it on good authority that the primary points of failure upon launch were related to middleware connecting the modules that were in fact using RDBMS technology.
Thirdly, many Slashdotters will be learning of MarkLogic Server for the first time through this article, and I urge them to give the technology a go before making a judgment call. For one thing, while XML is the (apparent) native storage format and XQuery is the native language, the actuality is somewhat different from what might be assumed. For one thing, the native storage format is not raw XML, but a fully-indexed compressed format which provides a decent compromise between storage requirements and rapid query/retrieval. Additionally, while XQuery is still the native language used by the technology, there has been a significant effort to provide a usable interface to the data through REsT, native Java and even basic SQL. Storage is journaled, implemented as an MVCC system and ACID-compliant in a way that no other "NoSQL" platform can offer. As an example of the platform's resilience under load, I am reliably informed that the BBC's real-time online coverage of the 2012 London Olympiad (which left other news and broadcast organisations in the dust) was powered by MarkLogic Server.
Just like Linux and the OSS BSD implementations that were creaming Microsoft's NT server implementations at the turn of the millennium, alternative data storage, query and search technologies are now challenging the old guard, and the old guard are running scared, and I'd bet significant money that the NYT sources are from the established RDMS vendors. I've been contracting with employers who have been using MarkLogic Server for the best part of a decade, and IMO the technology represents the most viable threat to the RDBMS hegemony and the vendors that rely on it that has ever existed. This opinion is not based on hype, it's based on nearly ten years of experience with the technology and fourteen years of experience with RDBMS development strategies.
I'm not preaching, in fact I urge you all to make your own minds up.
- "How do we do it? Volume!" - The Bursar of Unseen University.