Slashdot Mirror
NYT: Healthcare.gov Project Chaos Due Partly To Unorthodox Database Choice
First time accepted submitter conoviator writes "The NY Times has just published a piece providing more background on the healthcare.gov software project. One interesting aspect: 'Another sore point was the Medicare agency's decision to use database software, from a company called MarkLogic, that managed the data differently from systems by companies like IBM, Microsoft and Oracle. CGI officials argued that it would slow work because it was too unfamiliar. Government officials disagreed, and its configuration remains a serious problem.'" The story does not say that MarkLogic's software is bad in itself, only that the choice meant increased complexity on the project.
A little googling turns up that MarkLogic's offering is NoSQL. Without getting into the whole SQL/NoSQL debate, I can't help but noting that this is clearly relational data with a fairly limited number of records (clearly there can't be more than 300M people listed) and for which ACID is (or should be) a major concern.
"Some people, when confronted with a problem, think 'I know, I'll use XML.' Now they have two problems."
-JWZ
MarkLogic is an XML database, not a relational database, so if your data primarily consists of XML content then it's the right tool for the job. Sounds like the vendor building the system had a favorite hammer and decided that a rather traditional database problem looked like a nail.
MarkLogic itself is fine if your data fits neatly into an XML schema, but with healthcare.gov that tree is probably enormous and hard to optimize for DB activity.
Software Shouldn't Suck
E-mail: frank at jacquette dot spamless com (remove the spamless!)
I worked as a contractor developing a system at FDA. It lasted for 5 years. Inside the Beltway, it's pretty much the same all over. Dysfunctional communications and ridiculous breakdown of authority not corresponding to lines of management. No accountability. Project management requirements that have never been followed by any project. No commitment to the output of requirements gathering. No requirements change control. No performance engineering. Inadequate testing. No acceptance process by the government. IT groups with oversight for contractor output that have never written a line of code. All in all, pretty sick and ugly. Prior to my project there were 5 failed attempts. My project followed PMI practices, worked them hard and succeeded.
I ran into this recently at a company whose new head of engineering was talking to me (an outsider) about the technology problem they had to solve and I thought it sounded very traditional and simple except they'd need to carefully plan for horizontal scaling.
Basically a potentially huge number of devices (in the range of millions) would be reporting in periodic data that had to be stored and potentially evaluated in real-time. The data was quite easily swim laned by geolocation and the data had no appreciable inter-related significance. So basically, one piece of data from one device had nothing to do with any other device's information except in the general sense that can come from a more heuristic correlation of their data.
I should mention that the new engineering head and I had already (together) handle a situation very similar to this at a previously successful software company.
Well, the new engineering head had inherited an external architect who had different ideas. All of these different ideas involved things like Cassandra over Hadoop, AMP/Spark, BDAS. He showed me a diagram of the technologies he wanted to integrate and I'd never heard of almost half of them (and I deal with scaling issues all the time), this diagram had about 15 different technologies stacked together. It was crazy - all to solve a relatively simple data volume problem.
Almost needless to say, I advised otherwise, and afaik they're going the bid data way because it will make it easier for them to pull in VC money since shockingly few VC's actually evaluate technology before they put money in (I do this for VCs also, and other VCs wonder how I get paid to do this, lol.)
Loading...
Also, the Federal Government fscking LOVES XML. Like, a lot. The things I saw in the new "protocols" they're deploying for internatl air-traffic control data stuff... Wow...
Maxim: People cannot follow directions.
Increases in truth directly with the length of time spent explaining them
The chart's hosted on the National Republican Congressional Committee's website. I would take it with a heaping tablespoon of salt, if I were you. It's say to say that chart was designed to look as scary and confusing as possible.