Slashdot Mirror
Passport Database Outage Leaves Thousands Stranded
linuxwrangler (582055) writes Job interviews missed, work and wedding plans disrupted, children unable to fly home with their adoptive parents. All this disruption is due to a outage involving the passport and visa processing database at the U.S. State Department. The problems have been ongoing since July 19 and the best estimate for repair is "soon."
The system "crashed shortly after maintenance."
It's the wave of the future. A typical contract with offshore IT is for "current minus one", which means that each new firmware, OS or driver release causes a flurry of "maintenance" by remote "admins" who follow written procedures to update the systems with no real understanding of what they're doing, in what order they should do it, or what to do if something goes wrong. A typical list of systems to update may randomly contain a haphazard collection of prod and development machines, and may include some but not all members of a cluster. Systems are patched in Asset Management order, with no thought to rolling through dev and QA first before doing prod.
The backout plan is to engage the vendor.
Our outsourced IT bricks a few servers a year. We try to take it in stride. We've argued hysterically that if they really have to do firmware updates, to at least do dev servers first for God's Sake. They seem to not understand this.
So yeah, I could definitely see this happening. We will be seeing more of same. You get what you pay for.
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.