DoD Ditches Open Source Medical Records System In $4.3B Contract

dmr001 writes: The US Department of Defense opted not to use the Department of Veterans Affairs' open source VistA electronic health record system in its project to overhaul its legacy systems, instead opting for a consortium of Cerner, Leidos and Accenture. The initial $4.3 billion implementation is expected to be the first part of a $9 billion dollar project. The Under Secretary for Acquisition stated they wanted a system with minimum modifications and interoperability with private sector systems, though much of what passes for inter-vendor operability in the marketplace is more aspirational than operable. The DoD aims to start implementation at 8 sites in the Pacific Northwest by the end of 2016, noting that "legacy systems are eating us alive in terms of support and maintenance," consuming 95% of the Military Health Systems IT budget.

Comments

A consortium of Cerner, Leidos and Accenture

[Anonymous Coward]

Jesus Christ what a waste of money and to the worst possible people.

Re:A consortium of Cerner, Leidos and Accenture

[invictusvoyd]

We are deeply grateful to Source Forge for providing us with our place in cyberspace for this web site.

http://worldvista.org/AboutVis...
Agreed, but this sounds kinda weird too

Re:A consortium of Cerner, Leidos and Accenture

[arglebargle_xiv]

A large-scale medical records system, a multibillion-dollar IT project, and companies like Accenture doing it, it's like combining herpes, syphillis, and gonorrhea and hoping you'll get a cure for cancer. Any of of those in isolation is pretty much pre-ordained to fail, and they're combing them all into one massive clusterfsck... why don't they just declare failure in advance and save the years of effort (and money).

Re:A consortium of Cerner, Leidos and Accenture

[Anonymous Coward]

You obviously don't know anything about MUMPS.

I'll just discuss these specific points:
- It *is* a database.
- The "multidimensional array" is some sort of b-tree, the same kind of approach that is used by an RDBMS.
- The language is dynamically typed ("no ints or floats"), which in practice means it's like working in Python or Perl. Internally, numerical data is stored in a numerical format for efficiency.
- There's no need for reserved names, that's correct. Nonetheless, coding practices dictate that you don't name a variable "if" or something silly like that.
- There is a schema.
- Yes, a "global array" is a database object so of course it's stored to disk.
- "All program code is stored and executed within the global array alongside production data." It's stored in a specific location (outside the database schema) that's dedicated to program code. These are basically stored procedures. I imagine every database you're familiar with also keeps its stores procedures in the database.

I could educate you more thoroughly, but it would be a waste of my time.

Re:A consortium of Cerner, Leidos and Accenture

[JustAnotherOldGuy]

So...you're saying it's webscale? Thank god, because we were just about to do it the old-fashioned way so we would end up with a working product before the end of the century.

Trading one for the other

[grilled-cheese]

It's interesting how they see integrating legacy systems any differently integrating just as many differently implemented commercial record systems. The data integrators will make the same money either way. By abandoning the open-source solution, you're just losing the possibility others might benefit from the work. Likewise, I'm curious how much those 3 vendors have lobbied in Washington DC.

Re:Trading one for the other

[drooling-dog]

I have no knowledge of the particulars in this case, but lobbying isn't even really necessary. It's often just the revolving door: The procurement people on the government side now have very lucrative careers in the private sector to look forward to, and that is something you can never get by going with the open source solution. But who knows, maybe this time they did make the call purely on its technical merits.

Re:Trading one for the other

[Anonymous Coward]

The open source solution was not a good one in this case. You can have religious wars all you want about which language is best, but when striving for worst, MUMPS is a real contender. Please don't copy the database structure of VISTA. It's utterly useless for data integrity -- imagine a database where every field is a string. There are no numeric fields...and what happens? Oh.

The biggest problem is that the DoD lacks the organizational and technical skills at higher levels to use anything other than a defacto standard. They need something akin to RFC's, but that is not the military way.

The $9B price tag is actually reasonable for a good implementation and rollout. We know history too well: no one will see anything from the project for 10 years, and it will be rolled out on top of a lackluster Oracle database and dozens of middleware pieces that will ensure the system remains more expensive than current operation costs forever. And it will probably be less functional that what it replaces.

It's a shame that such big consumer lacks the power to make a difference in the market by developing and promoting good standards.

Yes, I'm bitching. I've been in the system long enough to lose hope.

Re:Trading one for the other

[phantomfive]

Regardless of whether they are starting with open source software, or closed source software........if I ever paid $4.3 billion for some software, I guarantee I would be getting the source for it. If the government pays that much for a system, one of the requirements should be that it ends up open source.

Re:Trading one for the other

[Fire_Wraith]

It's not necessarily even corrupt, for that matter. It's about having the personal relationships, and moreover, knowing how the labyrinthine mess that is the DoD Acquisition process works. The rules are intended to keep it fair, but at the same time, also wind up pricing a lot of the inexperienced sorts out of the process simply because you have to know what language to use, how to structure it, etc.

follow the money

[Lead Butthead]

follow the money and the answer is in front of you.

$4.3 billion == guaranteed failure.

[serviscope_minor]

As far as I can tell, any IT project costing a billion or more is 100% guaranteed to fail.

Also, it sounds like they decided to source IT from Lufier, Mephistopheles and Satan, which incidentally also guarantees it to fail.

Re:$4.3 billion == guaranteed failure.

[wonkavader]

Tom DeMarco talks about the air traffic control software project in one of his books. The description of the hopeless situation in that case supports your idea.

I think when you have a lot of people's butts on the line and so failure is not an option but stagnation IS, what we would perceive as failure is almost certainly coming. You can retire without any fallout so long as you make sure nothing happens for 15 years. It's easy to do: Just make the specs vague, self-contradictory, and long. Very, very, long.

The project won't fail, but it won't succeed either. And you're safe, which is all that matters.

They would do much better to set up a few small teams and have them compete to build something with enough in common so one can be replaced by the other. And starting with the open source base would make sense there.

Re:$4.3 billion == guaranteed failure.

[dinfinity]

As far as I can tell, any IT project costing a billion or more is 100% guaranteed to fail.

No kidding.

If you pay everybody $200 000 per year, that equates to 21 500 man-years (!) of work. I don't know what kind of problems in record keeping they're going to solve, but for that kind of money it'd better involve employees doing that in gold plated jets flown by an artificially engineered unicorn that continually snorts prime-grade cocaine.

Accenture?

[Anonymous Coward]

Project already failed.

UK NHS

[martin]

Sounds alot like the disasterous Nhs epr systems. After 20 years weve had progress but the number suppliers is down 1 and a major cash sinkhole

Re:UK NHS

[dmr001]

I thought about the NHS program when posting this, described as "the biggest IT failure ever seen". After £10 billion+ was spent, Her Majesty's government largely abandoned the effort, though the linked article notes Computer Sciences Corporation declaring victory as 3 of 220 NHS trusts managed to use portions of the system. I first heard this story a couple of years ago on a shuttle bus to the headquarters of a large privately held EMR vendor in Wisconsin, when I noticed the accents around me weren't American (like me). I was sitting amongst a group of friendly pharmacists from Oxfordshire. They were going to adopt this proprietary system for their NHS trust (ignoring, I suppose, the large chunk of it that dealt with billing).

Besides the air of defeat of all those pounds sterling going down a lot of oddly designed British toilets, they had given up on the idea of interoperability with the systems of other NHS trusts adopting different systems from other proprietary vendors. Back in the US, we have all kinds of government prodding to promote interoperability and many self-congratulatory health IT standards organizations that have national meetings in sunny placed. But, the farthest we've got with inter-vendor communication in my medical office after 3 years of promises and finger-pointing is faxing documents to an image server from the speciality clinic 100 feet away into inscrutably named files. Then, I can hand transcribe the important bits by hand about my patient's heart conditions and colon tumors in order to have a hope of retrieving that information again when I need it.

Re:UK NHS

[Dr_Barnowl]

We do have interoperability standards : I used to work for the department that specifies them.

They just don't follow them. Case in point : the meta-standard (HL7 v3) that we used for our messaging had a mechanism for not just sending NULL values, but also sending a reason why they were null. (e.g. - the value wasn't measured, etc). The vendor had no truck with that, and was using magic numbers instead (e.g. baby weights of 9999g which is outside the realms of sanity for a newborn). I was tasked with revamping one of the messages. I specified that the proper NULL flavours get used and ditch the magic numbers.

The vendor at this point rolled out the "full system test" clause in their contract, whereby they could charge £N * 10^6 to perform a full system test, because we'd changed the behaviour of one field. They got their way and kept their magic numbers. Other systems expecting messages that met with the conventions of the overall meta-standard for data now have an additional development cost to cope with those magic numbers.

This is the reason for the focus on interoperability over just having standard data structures - it lets vendors continue to use their own proprietary data schemes, and raises a barrier to new participants in the market, not only do you have to implement all the standard interfaces to interoperate, but you probably also have to design in a "quirks" layer to cope with each vendors *special* variations.

$4.3 billion

[lkcl]

wow fuck. imagine how much advancement in software libre could be had for $4.3 billion if the contract had been awared. hell, even 1% of that would make a big fucking difference. someone - such as the gnumed developers to take even one random example - could, with help, have developed a medical records system for ohhh i dunno... the U.S. Dept of Defense, with that kind of money. just to take a random example, y'ken.

Re:$4.3 billion

[gtall]

You have no idea of scale.

Re: $9 billion dollar project?

[snowgirl]

Paper doesn't scale to the level required. Trust me, I've attended presentations from ex-Googlers on the topic.

Especially, when the paper weighs so much that it started deforming a building...

Part of the problem

[satsuke]

That is part of the problem .. trying to design a drop in replacement that replicates the current functionality and interoperability with other systems.

With government especially, you have lists of exceptions and custom one-off code to get something working, that it becomes impracticable to replace it without an equal or additional number of exceptions.

It's the kind of system that benefits from a "flush it all away" mentality of defining new standards and sticking to them.

Accenture?

[msobkow]

Accenture? Better double that initial estimate to $18 billion, and count on it rising further. :(

not enough bribes by OSS

[dltaylor]

US Government purchasing "works" by the payment of bribes. Usually, these are not simply cash payments, but the opportunity for lucrative "consulting contracts" at the providing companies for senior Penagon and Civil Service officials after leaving government "service". I've seen it enough to know that saving money, at equal or better performance, will not get a government contract. Maybe, if enough congresscritters and/or senaturds are bribed with campaign contributions and/or honorariums, they'll push a deal one way or the other, but that rarely has anything to do with saving purchasing costs.

gnu project? really?

[Anonymous Coward]

And who, in the gnu community, is going to take on the responsibility for all the enterprise scale stuff that needs to be done. I can see lots of folks wanting to scratch their particular itch by coding up some piece, but who's going to do the architecture design, ride herd on the developers, etc.; make sure that the documentation gets done and is usable and readable (because, ya know, all those packages out on github and sourceforge are ever so well documented)..

I mean responsibility as in "be willing to stand up in front of Congress and explain your progress or lack thereof". I don't see a Linus or Theo or Eric or, gods forbid, Richard, filling that role.

Re:Accenture

[chipschap]

OK, years late, most promised feature not done/working, Budget 3x+ over and you will move to a proprietary commercial product with 100% lock in.
Sweet, yes would be fun to see who's back accounts/family members got hired to land this future failure in motion ;)

You left out the part where no senior executive service (SES) people get fired but middle-level people who actually tried to make it work get the blame. Meanwhile the Beltway Bandits make off with millions and hire the SES people for fat salaries as a reward for sending all the money their way.

VistA is a nightmare

[Gravis Zero]

i was interested in VistA and what all the fuss was about, so i decided to check it out. turns out the backend is nightmare code that would would swear was machine generated. after some investigation i found out it's MUMPS (Massachusetts General Hospital Utility Multi-Programming System) code. a lot of useful stuff started way back in the 1970s... but MUMPS is a 1960s nightmare come to life.

think i'm exagerating? here's a module from VistA's code which was apparently updated in 1989.

DENTA1 ;ISC2/SAW,HAG-DENTAL TREATMENT DATA SERVICE REPORTS ; 1/10/89 11:08 AM ; ;;1.2;DENTAL;**24**;JAN 26, 1989
  D:'$D(DT) DT^DICRW S %O="OPT",U="^",S=";",O=$T(@(%O)),DENTV=$$VERSION^XPDUTL("DENT") I $D(^DOPT($P(O,S,5),"VERSION")),(DENTV=^DOPT($P(O,S,5),"VERSION")) G IN
  K ^DOPT($P(O,S,5))
  F I=1:1 Q:$T(@(%O)+I)="" S ^DOPT($P(O,S,5),I,0)=$P($T(@(%O)+I),S,3),^DOPT($P(O,S,5),"B",$P($P($T(@(%O)+I),S,3),"^",1),I)=""
  S K=I-1,^DOPT($P(O,S,5),0)=$P(O,S,4)_U_1_U_K_U_K K I,K,X S ^DOPT($P(O,S,5),"VERSION")=DENTV
IN I $P(O,S,6)'="" D @($P(O,S,6))
PR S O=$T(@(%O)),S=";" S IOP=$I D ^%ZIS W:IOST'["PK-" @IOF K IOP
  I $P(O,S,7)'="" D @($P(O,S,7))
  E W !!,$P(O,S,3),":",!,$$VERSION^XPDUTL("DENT")," ",$P($T(+1),S,1),!!,$P(O,S,4),"S:",!
  F J=1:1 Q:'$D(^DOPT($P(O,S,5),J,0)) S K=$S(J0 S Z2=Z1
  G:Z3=0 W I Z3>1 S DIC="^DENT(225,",DIC(0)="AEMNQ",DIC("A")="Select STATION.DIVISION: " S:$D(DENTSTA) DIC("B")=$S(DENTSTA[" ":+DENTSTA,1:DENTSTA) D ^DIC Q:Y

Re:VistA is a nightmare

[Dutch Gun]

Wow... I mean... wow. This is the definition of a "write-only" language. I'm pretty sure you'd need external documentation just describing what these routines do, because it sure as hell looks like you're not going to derive it from this encryption disguised as source code. I thought perhaps you had chosen some particularly horrible section, like maybe it was a data definition of some sort. Nope, after sifting through a bunch of code, it pretty much all looked like that. My brain hurts just trying to parse and make sense of some of that code.

Is it any wonder that, with a language like this, it can't be easily extended and upgraded to meet demands? Yeah, okay, it's understandable now why they're tossing the whole thing. Open source or not, I can't imagine there are many people today who are able to extend or even maintain this monstrosity in perpetuity.