Passport Database Outage Leaves Thousands Stranded

linuxwrangler (582055) writes Job interviews missed, work and wedding plans disrupted, children unable to fly home with their adoptive parents. All this disruption is due to a outage involving the passport and visa processing database at the U.S. State Department. The problems have been ongoing since July 19 and the best estimate for repair is "soon." The system "crashed shortly after maintenance."

Comments

Change management fail

[dave562]

Rollback plan? What is that?

Re:Change management fail

[roc97007]

It's the wave of the future. A typical contract with offshore IT is for "current minus one", which means that each new firmware, OS or driver release causes a flurry of "maintenance" by remote "admins" who follow written procedures to update the systems with no real understanding of what they're doing, in what order they should do it, or what to do if something goes wrong. A typical list of systems to update may randomly contain a haphazard collection of prod and development machines, and may include some but not all members of a cluster. Systems are patched in Asset Management order, with no thought to rolling through dev and QA first before doing prod.

The backout plan is to engage the vendor.

Our outsourced IT bricks a few servers a year. We try to take it in stride. We've argued hysterically that if they really have to do firmware updates, to at least do dev servers first for God's Sake. They seem to not understand this.

So yeah, I could definitely see this happening. We will be seeing more of same. You get what you pay for.

Re:Change management fail

[Anonymous Coward]

Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no". The answer is always "yes" or some other affirmative that makes you think they have it under control and can do the work. When the fact is, the work they just said "yes" to, they don't actually have a clue how to perform it, so they learn as they go, on your production servers. They don't know what development / test environments are.

Re:Change management fail

[ShanghaiBill]

Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".

It takes two to fail to communicate. You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.

Re:Change management fail

[ShanghaiBill]

Sorry, what part of paying you to do a job requires me to give a shit about whether or not your failed third-world culture doesn't like answering direct fucking questions?

The part about you paying them far less than you would pay someone culturally compatible. If you want to pay peanuts, you need to deal with the cultural consequences. I have dealt with Indians for years, and have learned how to ask questions so that I get the answer I am looking for. It is not that hard.

Re:Change management fail

[Areyoukiddingme]

Well there's your problem! God has no part in an IT management plan.

Yeah, the other guy has it well in hand.

Re:Change management fail

[khchung]

Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".

On the other hand, I have encountered plenty of managers who literally do not know how to take "no" as an answer.

Takes two to make a pair.

Re:Change management fail

[ShanghaiBill]

simple yes or no questions

It is only simple because you speak English. You need to widen your cultural perspectives. In other languages, and other cultures, it is not so simple. For instance, Chinese does not even have the words "yes" and "no". If you ask a Chinese speaker if they have a pen, they will answer "have" or "not have". If you ask them if they are going to lunch, they will answer "going" or "not going". There is no such thing as a "yes or no question" in Chinese, and culturally, Chinese are much more direct than Indians or Japanese.

Re:Change management fail

[ruir]

Well, it also affects my ability to do the job people lying to me or choosing to reply with half truths to save face. My culture considers that extremely rude too. The rules of engagement have to change in a multicultural world, and if I am the customer, their obligation to bend somewhat their culture is a ball on their side. Or I may take my business and wallet elsewhere.

The solution

[Loopy]

Sic the healthcare.gov guys on it. I'm sure it'll be right as rain in no time.

Replication != Backup

[Anonymous Coward]

From their Q&A:

Q: Why wasn’t there a back-up server?
Back-up capability and redundancy are built into the system. The upgrade affected our current processing capability, in part because it interfered with the smooth interoperability of redundant nodes.

We don't need backups, the data is replicated, we're cool.

Re:Replication != Backup

[sumdumass]

http://xkcd.com/327/

I wonder if little bobby tables applied for a passport while this was going on?

Ask the NSA

[Daemonik]

I'm sure they have full copies of all the data already.

Large Databases?

[TechyImmigrant]

The article tries to wow us with the hugeness of the database, like this is a reason for the issues.

Yet the numbers quoted are not that big. Any modern PC isn't going to get too upset handling 75 million things. A real data center is going to sit there wondering what to do with the remaining 500TB of storage.

I don't doubt that there is some horrible flaw in the way the system was conceived that rendered it fragile, but whatever it is, it's nothing to do with the enormity of the problem, because it isn't very enormous.

Re:Here's the problem...

[AJWM]

Or worse, they're running SQLServer on Sun boxes...