Slashdot Mirror
Passport Database Outage Leaves Thousands Stranded
linuxwrangler (582055) writes Job interviews missed, work and wedding plans disrupted, children unable to fly home with their adoptive parents. All this disruption is due to a outage involving the passport and visa processing database at the U.S. State Department. The problems have been ongoing since July 19 and the best estimate for repair is "soon."
The system "crashed shortly after maintenance."
Rollback plan? What is that?
They are hard drive experts!
"I say we take off, nuke the site from orbit. It's the only way to be sure."
Still, bet Sysadmin's the highest ranking head that'll roll.
Happiness in intelligent people is the rarest thing I know.
Ernest Hemingway
Sic the healthcare.gov guys on it. I'm sure it'll be right as rain in no time.
http://happyplace.someecards.com/confession/elevator-work-in-progress-funny-sign/
From their Q&A:
Q: Why wasn’t there a back-up server?
Back-up capability and redundancy are built into the system. The upgrade affected our current processing capability, in part because it interfered with the smooth interoperability of redundant nodes.
We don't need backups, the data is replicated, we're cool.
We call this being over improved. So much for testing.
I hope this caused some synapses to fire.
One Database to bind them.
One Database to keep them out.
And into the darkness send them.
I'm sure they have full copies of all the data already.
Glad I'm not 'murrican. With the inmates firmly in control of the asylum I'm inclined to listen to the tinfoilhatters who think it's a plot to control the populace.
That these breakdowns are lame excuses. If computers fails, have people forgot how to do the same process manually? It is better to halt all the flights than letting people through and risk "terrorists" flying? Are we that terrified?
The whole US customs and immigration system is massively dysfunctional. Last year I flew into Minneapolis from Asia. I'd been traveling for twenty hours straight and then I got to stand in line for a full hour waiting for an immigration agent to spend ten seconds looking at my passport photo to make sure it matched my face. Even the third world airports I've been through aren't that bad. There were even empty stations without agents. How much would it have cost to add a few more agents - $100? At the time they were doing this ridiculous upgrade to the airport that must have cost millions - they were setting up all these silly little tables with ipads in the waiting areas. But somehow they couldn't manage to have enough immigration agents. It made me wonder if people in the state of Minnesota are as silly as their ariport - they did elect Michelle Bachmann to congress - so there may be quite a few of them who were dropped on their heads as babies or something.
Watch as government system availability falls in ways previously deemded "unlikely". Currently, 97% at most (http://en.wikipedia.org/wiki/High_availability#Percentage_calculation) and perhaps approaching the 66% US government systems are known for. All they need is more money for more warm bodies to make better mistakes, otherwise it wouldn't be proper welfare to have a government job.
I think I found the problem, from the Department of State's own website:
"The Department of State is working with Oracle and Microsoft to implement system changes aimed at optimizing performance and addressing ongoing performance issues."
They're running Oracle on Windows.
I have arrived at the point where any crashes experienced by whatever State Department of whatever so called and self proclaimed Democratic Country (traitor mark here) are welcomed by me with the utmost glee. The more disruption, the more chances for a turnaround.
somewhere there was a choice made that was a poor one.
no back out plan.
no test environment.
poor choice of vendor (software or hardware )
bad application/database config/design
speaking as a system admin, my bet would be a management person, that was promoted.
And was promoted for getting this system installed under budget !
Then left the company for more money.
is this is something to do with Russia and Ukraine
And how many families will be disconnected because of this?
How many jobs will be lost when people can't get back to work?
It's all nice conveniently glossing over the fact that people can't get home but they have lives to live, schedules to meet and contracts their obligated to perform under. You can't just say "Oh, sorry. You can't come back. Try later." The real world doesn't accept "Try later" as an excuse.
Two reasons for this fuck up come to mind immediately:
The Department of State is working with Oracle and Microsoft to implement system changes aimed at optimizing performance and addressing ongoing performance issues.
Of course, no one would expect the State Department to use anything else, but given what they're running there should be little surprise that an update caused unexpected, catastrophic, inexplicable and seemingly irreparable performance issues.
http://www.homestarrunner.com/systemisdown.html
"The system 'crashed shortly after maintenance.'"
Payback for the DoJ's request of e-mails from a server in Dublin of a person of interest which bcc'ed Bill Gates.
M$ targeted the DoS servers and fead them malware and hack disguised as an Update.
The DoS's Visa and Passport databases have now been subversively transferred to Al Quida servers.
This will tie-up airline flights as every passenger on all US (out and in) flights must be co-verified on the DoS server databases through DHS.
Rough seas ahead.
The article tries to wow us with the hugeness of the database, like this is a reason for the issues.
Yet the numbers quoted are not that big. Any modern PC isn't going to get too upset handling 75 million things. A real data center is going to sit there wondering what to do with the remaining 500TB of storage.
I don't doubt that there is some horrible flaw in the way the system was conceived that rendered it fragile, but whatever it is, it's nothing to do with the enormity of the problem, because it isn't very enormous.
I should use this sig to advertise my book ISBN-13 : 978-1501515132.
remains completely open and unguarded.
Who let Lois Lerner in the State Department's office?
I understand that the database is very large by any measure: 100 million records, 75 million pictures. But social security databases in most countries (or income tax databases) are at least that large (ok, likely much larger). Its a fail to have a large database really tank like this. If you need to shut the whole thing down for a day to avoid corrupt data, then shut it down. Fixing a corrupt database is much more difficult than correctly shutting a (slow) one down and then bringing it back up again.
While it doesn't always go this way, often simple things like the User Experience of a business gives an indication of the ethos behind a whole lot of the processes and systems they are using. To wit, compare the US Arrivals card that all "aliens" need to use upon arrival into the US, with the one from Australia. A clear 1970s look-and-feel versus something from this millenium.
http://www.immihelp.com/visas/sample-i94-form.pdf
http://www.immi.gov.au/managing-australias-borders/border-security/travel/passenger-cards/_pdf/english-ipc-sample.pdf
Just fly over to Mexico and ....... well you can guess it from there!
Ultimately my example was about: "level of incompetence and lack of planning is strong in several levels", as you suggested but it was driven that way by the new vendor having far too much control over the situation and no risk to bear in the event of failure.
The government took them to court twice (outgoing and incoming - Queensland, Australia) and could not scratch that vendor (IBM) for any of the $500 million+ in estimated extra costs.
For another 840 million dollars they can probably get it to the point where only another 150 million is needed to get it running.