Slashdot Mirror


NYT: Healthcare.gov Project Chaos Due Partly To Unorthodox Database Choice

First time accepted submitter conoviator writes "The NY Times has just published a piece providing more background on the healthcare.gov software project. One interesting aspect: 'Another sore point was the Medicare agency's decision to use database software, from a company called MarkLogic, that managed the data differently from systems by companies like IBM, Microsoft and Oracle. CGI officials argued that it would slow work because it was too unfamiliar. Government officials disagreed, and its configuration remains a serious problem.'" The story does not say that MarkLogic's software is bad in itself, only that the choice meant increased complexity on the project.

31 of 334 comments (clear)

  1. follow the money by ganjadude · · Score: 5, Insightful

    Who owns this company?

    how much do they contribute to XXX???

    There has got to be some reason that this DB that ive never even heard of (and i work with DBs, its not my main point of work but I know my way around DBs) got the gig over the more established players.

    or, perhaps they went with it because it is less known and therefore reduce the risk of known attacks in other DB systems?

    --
    have you seen my sig? there are many others like it but none that are the same
    1. Re:follow the money by Anonymous Coward · · Score: 5, Insightful

      from wikipedia,

      The company’s main product, MarkLogic Server, is an XML based database management server.

      This sentence has basically summed up the entire problem.

    2. Re:follow the money by Desler · · Score: 5, Funny

      They should have just used MongoDB. I hear it's web scale. *ducks*

    3. Re: follow the money by MyDixieWrecked · · Score: 5, Insightful

      I used marklogic when I worked at a previous job and after learning how it worked and understanding it better, it made our jobs incredibly easy. It just had a serious learning curve.

      Marklogic is a nosql db, that uses XML for its object format and xquery for its query language. This thing is NOT mongodb. It actually works really well and allows for complex data modeling with the ability to do joins and have transactional isolation in making changes to the data as well as a really solid content processing framework with pipelining and all that jazz.

      Now, I can't imagine a reason for using marklogic, or any non-relational db for a project like this. The only clue is that marklogic has a lot of government contracts; mostly for the military. So maybe that's why it was used. But the fact that they chose a database system that they weren't experts in for a project that had so much visibility speaks volumes on how mismanaged this whole project was.

      --



      ...spike
      Ewwwwww, coconut...
    4. Re: follow the money by Nerdfest · · Score: 4, Insightful

      To me, the fact that it was a major problem is an indicator of bigger problems. With systems like this, the database should be just a simple repository for persisting an retrieving domain models. If they want to do any non-trivial reporting on it the data should be replicated to a reporting repository. Treating the database a a simple persistence repository makes most database operations trivial, and better yet, decouples the database from the business, meaning that if you couldn't even manage the simple operations, replacing the layer is fairly trivial.

      The problem is when people think the database is the starting point for a system, not just a tool to be used to support what your business logic is doing. It sounds like that's what happened here.

    5. Re: follow the money by Chris+Mattern · · Score: 4, Insightful

      In fact, all of those would suck donkey balls. That is because XML, JSON and XQuery are all *data-interchange formats*. Using them to run your database internals is like using a screwdriver as a hammer.

    6. Re:follow the money by mlts · · Score: 5, Insightful

      For them, it is a wise choice. It brings job security using a non mainstream RDBMS, which means that the data would take man-years to be migrated to Oracle, DB/2, or even MS SQL server which may not have the horsepower of a SPARC or POWER7 box, but one can always partition and cluster the RDBMS. It also is expensive to keep maintained, so it provides additional job security, as the expertise for this architecture is rare.

      Great win for the contracting company. Big loss for everyone else.

    7. Re:follow the money by Chibi+Merrow · · Score: 4, Interesting

      Also, the Federal Government fscking LOVES XML. Like, a lot. The things I saw in the new "protocols" they're deploying for internatl air-traffic control data stuff... Wow...

      --
      Maxim: People cannot follow directions.
      Increases in truth directly with the length of time spent explaining them
    8. Re:follow the money by PapayaSF · · Score: 5, Insightful

      It definitely explains the problems they've been having.

      But this is only part of it. Check out this diagram. I'm no expert, but look at all the government systems in the upper left that Healthcare.gov is supposed to communicate with, in real time. And on the right, 50 state Medicaid systems. And at the bottom, all the insurance companies. Far, far simpler IT projects have failed. This site will not be ready by the end of the month as promised, and there is a good chance that it will never work as planned.

      --
      Q: What does the "B." in Benoit B. Mandelbrot stand for? A: Benoit B. Mandelbrot
    9. Re: follow the money by Aryden · · Score: 4, Insightful

      The reason we are control freaks is because when some moron wants to make a change and breaks the whole system, we are the ones that catch the blame and have to try to fix it if we don't get shit canned for it. You ever tried to re-key millions of records across hundreds of tables? You ever had to spend 3 days restoring db's from backup and having a site outage for 3 days because of it?

    10. Re:follow the money by killkillkill · · Score: 4, Funny

      XML is like violence. If it doesn't solve your problems you are not using enough of it.

      I'm starting to think that saying came from the government.

    11. Re:follow the money by artor3 · · Score: 4, Interesting

      The chart's hosted on the National Republican Congressional Committee's website. I would take it with a heaping tablespoon of salt, if I were you. It's say to say that chart was designed to look as scary and confusing as possible.

    12. Re:follow the money by BlueStrat · · Score: 4, Informative

      I'm guessing they were lowest bid.

      Bids? What bids?

      There was no bidding.

      http://nypost.com/2013/11/01/obama-donors-firm-hired-to-fix-web-mess-it-helped-make/

      ----
      "WASHINGTON â" A tech firm linked to a campaign-donor crony of President Obama not only got the job to help build the federal health-insurance Web site â" but also is getting paid to fix it.

      Anthony Welters, a top campaign bundler for Obama and frequent White House guest, is the executive vice president of UnitedHealth Group, which owns the software company now at the center of the ObamaCare Web-site fiasco.

      UnitedHealth Group subsidiary Quality Software Services Inc. (QSSI), which built the data hub for the ObamaCare system, has been named the new general contractor in charge of repairing the glitch-plagued HealthCare.gov.

      Welters and his wife, Beatrice, have shoveled piles of cash into Obamaâ(TM)s campaign coffers and Âapparently reaped the rewards.

      Beatrice Welters bundled donations totaling between $200,000 and $500,000 for Obamaâ(TM)s campaign during the 2008 election Âcycle, according to campaign- Âfinance data compiled by Center for Responsive Politics.

      SICK CALL: Obama bundler Anthony Weltersâ(TM) firm owns the company picked to repair the health Web site.

      The couple then became top donors for Obamaâ(TM)s inauguration festivities, kicking in $100,000 out of their own pockets and bundling another $300,000 from friends and business associates, according to the center.

      The investments quickly paid off for Beatrice Welters. The Obama administration tapped her in 2009 for the plum job of US ambassador to Trinidad and Tobago, which she held through last November.

      The couple have been frequent guests at the White House.

      Visitors logs show at least a dozen visits between the two by the end of 2012, the most recent information available.

      The entire Welters family has gotten into the donation game.

      The Welters, along with their sons, Andrew and Bryant, have contributed more than $258,000 to mostly Democratic candidates and committees since 2007.

      Whatâ(TM)s more, UnitedHealth Group is one of the largest health-insurance companies in the country and spent millions lobbying for ObamaCare.

      The insurance giantâ(TM)s purchase of QSSI in 2012 raised eyebrows on Capitol Hill, but the tech firm nevertheless kept the job of building the data hub for the ObamaCare Web site where consumers buy the new mandatory health- Âinsurance plans.

      QSSI has been paid an estimated $150 million so far, but officials couldnâ(TM)t say how much more the company might collect on the Ârepair contract.

      By all accounts, the data hub has run smoothly while many other components of the Web site have failed.

      Meanwhile, tempers flared among Obamaâ(TM)s Democratic allies over the disastrous rollout of the presidentâ(TM)s signature initiative.

      âoeIâ(TM)m extraordinarily frustrated,â said Sen. Jeff Merkley (D-Ore.) Âafter top Obama-administration officials gave Senate Democrats a private briefing on the state of the Web-site repairs.

      He said they were losing confidence the site could be quickly fixed.

      âoeI donâ(TM)t think thereâ(TM)s confidence by anyone in the room. This is more of a show-me moment,â said Merkley."
      ----

      This thing was never meant to function in any event. They didn't pay attention to the details because they didn't matter.

      It was designed to fail so that the "Holy Grail" of single-payer government-run healthcare could be rolled out as a "fix". As a bonus, they get to hand a big bag of taxpayer's money to their "bag-man".

      Just watch. Single-payer will be the "fix" insisted upon.

      At least we won't have so many foreigners coming to the US for our excellent health care any more.

      Strat

      --
      Progressivism (aka US 'Liberalism'): Ideas so good they need a police/surveillance-state to enforce.
    13. Re:follow the money by Lord+Crc · · Score: 4, Funny

      They should have just used MongoDB. I hear it's web scale.

      Web scale or not, I have no idea. But I do think they've got the best bug report ever in their tracker.

    14. Re: follow the money by Aryden · · Score: 4, Insightful
      Changes to production that pass QA can still break the system once they are in the wild and every instance of execution begins. Additionally, not every change that goes into production in all instances goes through the DBA especially when there isn't a schema change. A dipshit writes an app that works fine in QA, migrates to production, due to a difference in version he didn't pay attention to, he overwrites 5 million records. It really does actually happen in the real world and they ( many companies) don't bother consulting the experts until it's already fucked up.

      Three days restoring is too long too. Unless your server is remote and upstream is super slow.

      This is becoming more commonplace as companies are outsourcing or contracting out their admins and development.

      But none of these things should give you permission to be a control freak. You are responsible for keeping the databases up, yes. But you are not responsible for everything that happens to them. Sometimes your dbs just get screwed up and you'll have to fix it. Just like sometimes business wants things done to the site/app that will totally jack up the fung shui i had going on in code. Nobody is purposefully trying to screw anyone over. We all want the business to grow and succeed. People make mistakes and shit happens. There's no need to also be an ass to each other on top of it all.

      You'd think that, but when it's your job on the line because fucktards in management have 0 clue and assume that because it has something to do with the database and you are the admin, it's your fault. And having been subject to intentional sabotage, and seeing it done numerous times, it exists and happens.

  2. Noobs? by Anonymous Coward · · Score: 5, Insightful

    FTA: "An initial assessment identified more than 600 hardware and software defects — 'the longest list anybody had ever seen,' one person involved with the project said. "

    Strikes me as none of these people seemed to have ever worked on large projects before.

  3. NIH syndrome by Gothmolly · · Score: 4, Insightful

    You could take a handful of proven DB technologies such as Oracle/DB/MSSQL, throw a web (Apache/IIS) and app (.Net/WAS/Jboss) front end to it, and it would work. Why did these guys fuck up the whole thing? It's like the scene in The Fountainhead when the second-rate architects smash up the plans and add their own stuff, "to express their own individuality". This could have been a solved problem - hell, it WAS a solved problem.

    --
    I want to delete my account but Slashdot doesn't allow it.
    1. Re:NIH syndrome by Okian+Warrior · · Score: 5, Insightful

      Why did these guys fuck up the whole thing?

      All actions by Federal government players seem rational when viewed from their point of view.

      The problem with most analysis, here or on other sites across the web, is in the assumptions. Specifically, if you assume that the government actions are for some benefit to the people, or efficiently completes some task (which is assumed to benefit the people), then nothing they do makes sense. That assumption makes for good outrage which can attract readers, but it doesn't answer the question of "why".

      When government actions are viewed in light of more obvious motivations, everything it does makes sense. You start to see them not as an pack of incompetent, bureaucratic screw-ups, but as a self-sustaining, self-absorbed, engine of corruption.

      I'm not aware of *anything* the federal government's done in the past two decades that's been in the interests of the people. All the useful stuff, the stuff you would want to keep across a reboot, was several decades ago.

      I don't know the specifics of why an obscure database was chosen over the obvious players, but the reason wasn't "best utility for the people". Look for another reason.

    2. Re:NIH syndrome by jythie · · Score: 5, Insightful

      Sad thing is, much of the behavior one sees out of federal contracts is due to taxpayer groups demanding anti-corruption measures. A great deal of the bureaucracy comes directly from people complaining about waste and demanding a complex audible process.

      I would actually put forward that the problem is not the government, our government is pathetically weak in many ways. The problem are all the private groups flexing their muscles and making the government dance. Most of the people I have worked with on these kinds of projects are good people who just want to get stuff done, but they are weighted down by a nearly impossible set of requirements, many of which are mutually exclusive, dropped on them by outside groups who mainly are interested in showing off their power to their own people and have no investment in the project itself being successful.... and often have an active interest in the project failing since they can always use that as fuel for more personal power.

  4. MarkLogic = NoSQL by NeverWorker1 · · Score: 4, Interesting

    A little googling turns up that MarkLogic's offering is NoSQL. Without getting into the whole SQL/NoSQL debate, I can't help but noting that this is clearly relational data with a fairly limited number of records (clearly there can't be more than 300M people listed) and for which ACID is (or should be) a major concern.

    1. Re:MarkLogic = NoSQL by Anonymous Coward · · Score: 5, Interesting

      A customer at work has a MarkLogic database. I don't know what version it is, but I assure you IT IS HORRIBLE. It's like... an XML database or some shit. Just awful, and extremely confusing to use.

      A couple years ago I was asked to do automatic database backups. The only documentation I could find for backing up the database requires logging into a web interface I had no idea existed on an obscure port and clicking a backup button. I literally had to write a perl script to fake a browser and do this just so they could get a database dump.

      Do not use this product. Please.

    2. Re: MarkLogic = NoSQL by MyDixieWrecked · · Score: 4, Interesting

      Marklogic, afaik, is the only acid compliant nosql solution that exists.

      --



      ...spike
      Ewwwwww, coconut...
  5. Blow to NoSQL movement by ilsaloving · · Score: 4, Insightful

    And here is a fantastic example of what happens when hype trumps common sense. NoSQL is the new hawtness, and apparently the dumbasses running the project wanted to be part of that. Now MarkLogic, and NoSQL in general, will have a massive blow to their reputations, and it's unknown how badly this will hurt them.

    As someone who has done databases for a long time, I have very little respect for NoSQL, but that is mostly because everyone keeps trumpeting it as a killer of traditional databases. There are scenarios where NoSQL systems are an ideal fit. However, NONE of those scenarios require data to be very reliably stored in a guaranteed and predictable way.

    If you don't get your tweets or your friends facebook posts as soon as they are posted, no one will really care. But for something as truly important as health insurance coverage? Are you f__king kidding me? And that's just from a reliability standpoint. Nevermind the fact that NoSQL is currently at the wild west stage where nobody is compatible with anybody else, there is nothing resembling a standard set of APIs between products, making it very difficult to develop expertise.

    Sounds like the Gov was just begging for problems.

    1. Re:Blow to NoSQL movement by Assmasher · · Score: 4, Interesting

      I ran into this recently at a company whose new head of engineering was talking to me (an outsider) about the technology problem they had to solve and I thought it sounded very traditional and simple except they'd need to carefully plan for horizontal scaling.

      Basically a potentially huge number of devices (in the range of millions) would be reporting in periodic data that had to be stored and potentially evaluated in real-time. The data was quite easily swim laned by geolocation and the data had no appreciable inter-related significance. So basically, one piece of data from one device had nothing to do with any other device's information except in the general sense that can come from a more heuristic correlation of their data.

      I should mention that the new engineering head and I had already (together) handle a situation very similar to this at a previously successful software company.

      Well, the new engineering head had inherited an external architect who had different ideas. All of these different ideas involved things like Cassandra over Hadoop, AMP/Spark, BDAS. He showed me a diagram of the technologies he wanted to integrate and I'd never heard of almost half of them (and I deal with scaling issues all the time), this diagram had about 15 different technologies stacked together. It was crazy - all to solve a relatively simple data volume problem.

      Almost needless to say, I advised otherwise, and afaik they're going the bid data way because it will make it easier for them to pull in VC money since shockingly few VC's actually evaluate technology before they put money in (I do this for VCs also, and other VCs wonder how I get paid to do this, lol.)

      --
      Loading...
  6. MarkLogic is an XML repository, not a RDBMS by CyberLeader · · Score: 5, Interesting

    "Some people, when confronted with a problem, think 'I know, I'll use XML.' Now they have two problems."

    -JWZ

    MarkLogic is an XML database, not a relational database, so if your data primarily consists of XML content then it's the right tool for the job. Sounds like the vendor building the system had a favorite hammer and decided that a rather traditional database problem looked like a nail.

    MarkLogic itself is fine if your data fits neatly into an XML schema, but with healthcare.gov that tree is probably enormous and hard to optimize for DB activity.

    --

    Software Shouldn't Suck

    E-mail: frank at jacquette dot spamless com (remove the spamless!)

    1. Re:MarkLogic is an XML repository, not a RDBMS by Tailhook · · Score: 4, Insightful

      Sounds like the vendor building the system had a favorite hammer and decided that a rather traditional database problem looked like a nail.

      It was the Medicare folks that imposed MarkLogic over the objections of the prime contractor. Calling Medicare a "vendor" is a bit of a stretch. Like non-IT staff calling themselves "customers" of IT in a corporation, as if they have a choice of IT departments.

      Another sore point was the Medicare agency’s decision to use database software, from a company called MarkLogic, that managed the data differently from systems by companies like IBM, Microsoft and Oracle. CGI officials argued that it would slow work because it was too unfamiliar. Government officials disagreed, and its configuration remains a serious problem.

      The CGI argument was that it would slow work. I have absolutely no doubt about that. Every NoSQL system is it's own distinct thing with unique APIs, protocols, conventions, quirks, etc. This is fine when you are dealing with a limited group of good developers that can be expected to rapidly climb a learning curve. This is not fine when you are the top contractor in a herd of contractors, populated by a vast number of mediocre programmers that last learned anything new in the late 90's.

      In that case the correct choice is to select the most ubiquitous technologies. A boutique NoSQL XML-base is not appropriate, even if it is excellent, which MarkLogic may be. I don't know. I just know CGI was 100% correct in objecting to it. CGI's fault in this case was its failure to win that argument.

      Not that it matters much. The project was doomed in any case — political agenda driving pie-in-the-sky requirements using crony contracting — Fucked at birth, as they say.

      --
      Maw! Fire up the karma burner!
  7. Re:Stop with the excuses. by DarkOx · · Score: 5, Insightful

    Bullshit. Since the law has been passed almost the entire implementation has been up to the administration. Which in case you had not noticed is lead by partisans in favor of the affordable care act.

    If the GOP can be accused of anything material in terms of interfering with the implementation it would preventing the law from being amended, and if it needs amending there is your proof it was bad law to begin with. Sorry Obama owns this and really HE owns it not even the larger DNC as its HIS branch of government that has been responsible for implementation from the moment HE signed the thing.

    *IF* it fails its either because it was unworkable in the first place as written and should not have been enacted or because the Obama administration failed to execute. Those are really the only honest high level conclusions.

    The argument well because they had to pass the Senate bill as is it did not get the usual fixes and tweaking is why its got so many problems is also bogus. They had to pass the Senate bill by abusing the budget reconciliation process to deny the minority party in the House at the time its rights; they knew if usual procedures were followed it would never have become law. So again it either should not have been enacted or the administration has failed to execute.

    --
    Repeal the 17th Amendment TODAY! Also Please Read http://www.gnu.org/philosophy/right-to-read.html
  8. Re:Open Source by NeverWorker1 · · Score: 4, Insightful

    Because Postgres exists.

  9. Re:Taxes... by Bengie · · Score: 4, Insightful

    The "47%" that people keep talking about are almost entirely retirees, adults going to full time college, or the disabled. The largest portion of the group are the retired people. Are you saying that people who have worked their whole like and are now retired, should not have a say in taxes?

  10. Re:Taxes... by Fwipp · · Score: 4, Informative

    Also the working poor - 47% pay no _income_ tax, but more than half of those people do pay payroll taxes, and even many those who don't (after deductions) still work.

    They also pay sales tax, property tax, etc etc.

  11. US Federal Government Systems Projects by vrhino · · Score: 5, Interesting

    I worked as a contractor developing a system at FDA. It lasted for 5 years. Inside the Beltway, it's pretty much the same all over. Dysfunctional communications and ridiculous breakdown of authority not corresponding to lines of management. No accountability. Project management requirements that have never been followed by any project. No commitment to the output of requirements gathering. No requirements change control. No performance engineering. Inadequate testing. No acceptance process by the government. IT groups with oversight for contractor output that have never written a line of code. All in all, pretty sick and ugly. Prior to my project there were 5 failed attempts. My project followed PMI practices, worked them hard and succeeded.