Passport Database Outage Leaves Thousands Stranded

linuxwrangler (582055) writes Job interviews missed, work and wedding plans disrupted, children unable to fly home with their adoptive parents. All this disruption is due to a outage involving the passport and visa processing database at the U.S. State Department. The problems have been ongoing since July 19 and the best estimate for repair is "soon." The system "crashed shortly after maintenance."

Comments

Change management fail

[dave562]

Rollback plan? What is that?

Re:Change management fail

[roc97007]

It's the wave of the future. A typical contract with offshore IT is for "current minus one", which means that each new firmware, OS or driver release causes a flurry of "maintenance" by remote "admins" who follow written procedures to update the systems with no real understanding of what they're doing, in what order they should do it, or what to do if something goes wrong. A typical list of systems to update may randomly contain a haphazard collection of prod and development machines, and may include some but not all members of a cluster. Systems are patched in Asset Management order, with no thought to rolling through dev and QA first before doing prod.

The backout plan is to engage the vendor.

Our outsourced IT bricks a few servers a year. We try to take it in stride. We've argued hysterically that if they really have to do firmware updates, to at least do dev servers first for God's Sake. They seem to not understand this.

So yeah, I could definitely see this happening. We will be seeing more of same. You get what you pay for.

Re:Change management fail

[Anonymous Coward]

Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no". The answer is always "yes" or some other affirmative that makes you think they have it under control and can do the work. When the fact is, the work they just said "yes" to, they don't actually have a clue how to perform it, so they learn as they go, on your production servers. They don't know what development / test environments are.

Re:Change management fail

[roc97007]

Oh man, don't get me started. It's not even clear that one would need to pay more -- we have not saved money so far by outsourcing, although the outsource company keeps telling us that savings are just around the corner. The first year, the excuse was that there is always startup issues, the second year, the excuse was that the outgoing employees did not document their jobs well enough, (probably true -- who would?) the third year the excuse was that the scope was bigger than we said it was. And so forth. Each year a new excuse and each year the total cost is more than what we were paying when we had our own IT department.

So yeah, insourcing, or at least selective insourcing, (let them keep doing what they do well, if anything) makes tremendous sense to me.

But I don't make the decisions.

And even where upper management has considered terminating our outsourcing contracts, it's only to give the contract to a different outsourcing company, which only means we're now calling a building across the street from the original building in Hyderabad. Who knows, we might even be dealing with some of the same people.

Re:Change management fail

[ShanghaiBill]

Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".

It takes two to fail to communicate. You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.

Re:Change management fail

[dave562]

As much as I am not a fan of government regulation, my professional experience has shown me that the only time people get IT anywhere close to right is when there is a risk of financial penalty involved in getting it wrong. Regulation seems to be the only solution to people working for peanuts. The people who work for peanuts make mistakes. If those mistakes cost the company more than the company saves by hiring those people, they will not hire those people.

Out of all of the industries that I have worked with, the financial services industries seem to be the most together. They are not perfect, but the penalties associated with losing customer data makes them more careful.

Re:Change management fail

[Anonymous Coward]

You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.

Do I care?

Re:Change management fail

[roc97007]

Exactly. Consider the business model -- A company tells you that you can outsource your IT department, buy it as a service, and pay the outsourcing company's overhead and significant profit margin, *and* save money. And the only way this could possibly work is if the outsourcing company goes to the LCC (least cost country) and hires the cheapest labor possible. This is justified in that all that IT stuff, it's all just following procedures, and anyone can do that.

And of course, this is a blatant falsehood, but executives of the victim company either (a) don't know that, (b) *want* to believe the scam, (like any good scam) or (c) don't care, because they intend to take their bonus and get the hell out of dodge.

So, outsourcing companies go into contracts *knowing* that cutover is going to be a Big Fail, and they have excuses prepared for when it happens. And a strategy (a brilliant strategy, really, executed by brilliant if unethical people) to string the con along as long as possible.

And just incidentally, the victim's attempts to train the workers that they are stuck with also fails in the long run, due to the nature of the business model. The business model breaks down if the workers are paid more than starvation wages, and workers with a bit of experience can easily get a higher paid job elsewhere (perhaps as a second or third level admin for a different outsourcing company) and they quit. And then you have a new person who doesn't know what a kernel is, and you have to train them up. And all you're doing is giving out free training. How noble of you.

...so not only is the system *designed* to fail, the system is very specifically designed to fail continuously.

But at least it's cheap. Oh, wait...

Re:Change management fail

[ShanghaiBill]

Sorry, what part of paying you to do a job requires me to give a shit about whether or not your failed third-world culture doesn't like answering direct fucking questions?

The part about you paying them far less than you would pay someone culturally compatible. If you want to pay peanuts, you need to deal with the cultural consequences. I have dealt with Indians for years, and have learned how to ask questions so that I get the answer I am looking for. It is not that hard.

Re:Change management fail

[Areyoukiddingme]

Well there's your problem! God has no part in an IT management plan.

Yeah, the other guy has it well in hand.

Re:Change management fail

[roc97007]

I sometimes think that if I accidentally entered a church with an IT management plan in a back pocket, my pants would burst into flames.

Re:Change management fail

[l0n3s0m3phr34k]

This isn't always the case. A company can save money via outsourcing IT infrustructure if they go with the right vendor. VMWare, virtual servers, proper fail-overs, big multi-core blade racks were the VM is still more powerful than your original server and still costs less...but of course I work at HP in the Enterprise Services so the level I'm talking about probably isn't affordable for a "small company". We have VERY specific steps for everything, our "runbooks" detail everything from server configs, hardware, rack enclosures, port layouts, and responsible parties to contact for each part if it fails. When you have a rack of blades, it's far easier to snapshot, launch then test,,,we always have a "backup" in a hot image ready to go if anything fails. Often I'm working with 5-15 people spread across the globe all doing different functions (unix admin, wintel, recovery, netops, etc) but we rarely have any "HP owned" customer-impacting outages. Of course my major clients are airlines so it's all tightly regulated; your individual milage may vary LOL.

Re:Change management fail

[khchung]

Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".

On the other hand, I have encountered plenty of managers who literally do not know how to take "no" as an answer.

Takes two to make a pair.

Re:Change management fail

[Anonymous Coward]

You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.

Do I care?

If affects your ability to do your job, you should.

Re:Change management fail

[JustOK]

Why? Are you a liar?

Re:Change management fail

[bayankaran]

Outsourcing is bad, but as an Indian who renewed his passport recently, I have to say the process was smooth.
The document verification/IT/software/hardware part of the Indian passport application/renewal operation is now handled by TCS. You make an appointment online - for both fast track and slow track - and arrive at the local passport center with your documents. The TCS grunts allot a token, make sure documents are in order - if they are not you are sent back to get them corrected, take your fingerprints, photo and for final verification you meet an Indian government employee. He/she either says YES, or says YES with police verification.
In my case it was the latter as my passport had some damage - usual wear and tear, but visible.
A few days later a policewoman came home after fixing an appointment over phone, verified my documents/address and in two weeks I got a fresh passport.
I was impressed by the whole process. AFAIK Indian government and TCS is doing everything right as far as passport renewal goes...it was better than getting my experience in getting a drivers license/immigration papers in Toronto/Chicago/San Francisco.
May be US government should indeed outsource the operation to TCS...if they can do a good show in an anarchic chaotic mess like India, I am sure they can do the same in US.

Re:Change management fail

[ShanghaiBill]

simple yes or no questions

It is only simple because you speak English. You need to widen your cultural perspectives. In other languages, and other cultures, it is not so simple. For instance, Chinese does not even have the words "yes" and "no". If you ask a Chinese speaker if they have a pen, they will answer "have" or "not have". If you ask them if they are going to lunch, they will answer "going" or "not going". There is no such thing as a "yes or no question" in Chinese, and culturally, Chinese are much more direct than Indians or Japanese.

Re:Change management fail

[hawkinspeter]

All of those are trivial to rollback. Just create a snapshot of the VM before the change and you can revert back to it if there's any problem.

Re:Change management fail

[ruir]

Well, it also affects my ability to do the job people lying to me or choosing to reply with half truths to save face. My culture considers that extremely rude too. The rules of engagement have to change in a multicultural world, and if I am the customer, their obligation to bend somewhat their culture is a ball on their side. Or I may take my business and wallet elsewhere.

Re:Change management fail

[ruir]

The question is that exactly because of that culture, they seem able to complicate the most simple of the problems looking through the eyes of our culture. It also does no help that they say they know everything, their CV looks like far better than a Linus Torvalds, lots of IT certifications, but when your start asking them the simplest of the questions, they fall apart.

Re:Change management fail

[Jesrad]

Sounds like your IT has been outsourced to India

Not necessarily. I've seen this exact kind of madness happen just as easily with locals, here in France. Like that time the local, on-site support team from our vendor rebooted the production server instead of the test platform, because woops wrong terminal window in the foreground.

Or when they covertly rolled out a "shame-bug fix" remotely on the production platform during a week-end night, again instead of targetting the test platform, then noticed their mistake, and wiped-out months of production data by reverting to a long-expired backup.

Or when the local datacenter people managed to botch our fully-automatized install+deploy+configure solution by messing up on the one thing they had to do right - that is, upload it and launch it on the correct machine of the cluster.

Don't think hiring local people for more money protects you from such cringe-worthy nonsense. The moment you outsource anything, and I do mean *anything*, no matter how far and how expensive and what nationality: if you base your expectations on anything but an actual track-record of reliability and dependability, you're exposing yourself to long hours of hair-pulling and yelling into phones.

Re:Change management fail

[Dr. Gamera]

Appreciate your comment. Can you provide some examples of how you would word questions to get a useful answer?

Re:Change management fail

[dave420]

You use a second machine, doofus. It's not rocket science. Mirror A to B, make changes to B, switch to B, if B breaks, switch back to A, if B doesn't break, you're golden.

Re:Change management fail

[phorm]

Yes=en. Its a short syllable pronounced from the throat with the mouth closed (somewhat grunt-like but not so inelegant). This is effective for most interrogatives.

No is roughly "bu" (boo). This is more a negative so it is often combine with other words, but "bu xie" (Boo shie / no thank you) is generally effective, but in the case of the "do you have" question a straight "boo" wouldn't be correct and the "I don't have" variety is useful. Conversely, "do you want [my pen/to go for lunch]" is generally answerable with "bu xie"

Maybe the parent should swap "simple positive and negative" instead of "yes and no", but the intent is the same and quite often it's not a language issue are you've put forth so much as a culture issue in terms of providing a straightforward response (whether it be ethnic culture or industry culture, etc).

It quite often still comes down to certain people or groups (companies etc) just being naturally evasive towards a straightforward answer.

The solution

[Loopy]

Sic the healthcare.gov guys on it. I'm sure it'll be right as rain in no time.

Replication != Backup

[Anonymous Coward]

From their Q&A:

Q: Why wasn’t there a back-up server?
Back-up capability and redundancy are built into the system. The upgrade affected our current processing capability, in part because it interfered with the smooth interoperability of redundant nodes.

We don't need backups, the data is replicated, we're cool.

Re:Replication != Backup

[sumdumass]

http://xkcd.com/327/

I wonder if little bobby tables applied for a passport while this was going on?

Ask the NSA

[Daemonik]

I'm sure they have full copies of all the data already.

I wonder

[ruir]

That these breakdowns are lame excuses. If computers fails, have people forgot how to do the same process manually? It is better to halt all the flights than letting people through and risk "terrorists" flying? Are we that terrified?

Re:I wonder

[Kittenman]

That these breakdowns are lame excuses. If computers fails, have people forgot how to do the same process manually? It is better to halt all the flights than letting people through and risk "terrorists" flying? Are we that terrified?

You could just ask the questions that used to get asked back in the '50s. "Do you intend to bring down or otherwise defame the US government?"

Really. A UK humourist (Frank Muir?) wrote "Sole purpose of visit" on the form.

Not just the passports

[Anonymous Coward]

The whole US customs and immigration system is massively dysfunctional. Last year I flew into Minneapolis from Asia. I'd been traveling for twenty hours straight and then I got to stand in line for a full hour waiting for an immigration agent to spend ten seconds looking at my passport photo to make sure it matched my face. Even the third world airports I've been through aren't that bad. There were even empty stations without agents. How much would it have cost to add a few more agents - $100? At the time they were doing this ridiculous upgrade to the airport that must have cost millions - they were setting up all these silly little tables with ipads in the waiting areas. But somehow they couldn't manage to have enough immigration agents. It made me wonder if people in the state of Minnesota are as silly as their ariport - they did elect Michelle Bachmann to congress - so there may be quite a few of them who were dropped on their heads as babies or something.

Here's the problem...

[Anonymous Coward]

I think I found the problem, from the Department of State's own website:

"The Department of State is working with Oracle and Microsoft to implement system changes aimed at optimizing performance and addressing ongoing performance issues."

They're running Oracle on Windows.

Re:Here's the problem...

[AJWM]

Or worse, they're running SQLServer on Sun boxes...

Large Databases?

[TechyImmigrant]

The article tries to wow us with the hugeness of the database, like this is a reason for the issues.

Yet the numbers quoted are not that big. Any modern PC isn't going to get too upset handling 75 million things. A real data center is going to sit there wondering what to do with the remaining 500TB of storage.

I don't doubt that there is some horrible flaw in the way the system was conceived that rendered it fragile, but whatever it is, it's nothing to do with the enormity of the problem, because it isn't very enormous.