Anatomy of the VA's IT Meltdown

Lucas123 writes "According to a Computerworld story, a relatively simple breakdown in communications led to a day-long systems outage within the VA's medical centers. The ultimate result of the outage: the cancellation of a project to centralize IT systems at more than 150 medical facilities into four regional data processing centers. The shutdown 'left months of work to recover data to update the medical records of thousands of veterans. The procedural failure also exposed a common problem in IT transformation efforts: Fault lines appear when management reporting shifts from local to regional.'"

Comments

Cache is part of the problem

[FBodyJim]

I work for a company that uses the Intersystems Cache database and I have to say that I imagine that Cache is a large part of the problem. The amount of good documentation for Cache lies between very little and none and my company has been on a nationwide search for people experienced with Cache and they too seem few and far between. Of course, I don't know that Cache really is a "worse" or "better" database that Oracle, SQL Server or MySQL for that matter, however, what I do know that is when it comes to experience, common tasks, documentation, examples and just getting things done, Cache lags far behind the others, not to mention Universities are still teaching relational db theory, not object db theory, at least when I graduated Rutgers a few short years ago. I suspect that given the task of merging databases, even large databases, there are plenty of experienced and knowledge SQL Server, Oracle, mySQL guys out on Monster or some other job site that know how to get the job done, efficiently and correctly, and have done the job a few times before. Based on our current and past searches for people capable of even easier tasks within Cache, there aren't many people out there with any Cache experience, never mind good people with Cache experience, and it's easy to fudge a task when you aren't given much good documentation, examples or experience. In a past career, I worked for a healthcare company that used SQL server for electronic medical records (EMRs) and the system worked rather well. There might have been better ways to design the database, stored procs or application code, however, we never had a problem hiring good staff that understood the database design, SQL queries, T-SQL/stored procs and as i said, I can't say the same about trying to hire good people who know and understand Mumps ("M" the language, not the disease) or Cache ObjectScript or find the Cache tools to be easy and intuitive. Just my $.02, and I don't mean to start a DB debate, just stating that it might just also be time for the VA's to move on from MUMPS/Cache to a more widely used and documented database and programming language, find some new blood.

Re:It happens

[Critical Facilities]

because some IT staffer changed a port # at one of their hub data centers without following proper procedure -- that's minor.
I don't know if I agree with that. "Change Control" or "Change Management" is a crucial part of any Data Center. The fact that these ports were changed without being properly "run up the flagpole" is a glaring mistake with very unfortunate results. I'll bet anyone swapping ports in the future will ask permission several times over before trying it again.