Passport Database Outage Leaves Thousands Stranded

← Back to Stories (view on slashdot.org)

Passport Database Outage Leaves Thousands Stranded

Posted by Unknown on Thursday July 31, 2014 @10:54AM from the maintenance-considered-harmful dept.

linuxwrangler (582055) writes Job interviews missed, work and wedding plans disrupted, children unable to fly home with their adoptive parents. All this disruption is due to a outage involving the passport and visa processing database at the U.S. State Department. The problems have been ongoing since July 19 and the best estimate for repair is "soon." The system "crashed shortly after maintenance."

115 of 162 comments (clear)

Min score:

Reason:

Sort:

Change management fail by dave562 · 2014-07-31 10:57 · Score: 4, Insightful

Rollback plan? What is that?
1. Re:Change management fail by Fallen+Kell · 2014-07-31 11:03 · Score: 1
  
  Yeah I was going to say the same thing. You NEVER make a change that you don't have a way to backout to the previous operational state.
  
  --
  We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
2. Re:Change management fail by roc97007 · 2014-07-31 11:15 · Score: 5, Informative
  
  It's the wave of the future. A typical contract with offshore IT is for "current minus one", which means that each new firmware, OS or driver release causes a flurry of "maintenance" by remote "admins" who follow written procedures to update the systems with no real understanding of what they're doing, in what order they should do it, or what to do if something goes wrong. A typical list of systems to update may randomly contain a haphazard collection of prod and development machines, and may include some but not all members of a cluster. Systems are patched in Asset Management order, with no thought to rolling through dev and QA first before doing prod.
  The backout plan is to engage the vendor.
  Our outsourced IT bricks a few servers a year. We try to take it in stride. We've argued hysterically that if they really have to do firmware updates, to at least do dev servers first for God's Sake. They seem to not understand this.
  So yeah, I could definitely see this happening. We will be seeing more of same. You get what you pay for.
  
  --
  Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
3. Re:Change management fail by roc97007 · 2014-07-31 11:17 · Score: 1
  
  ...in case my other article did not make it clear, we always ask if they have a backout plan, and they always say they do.
  ...and then, when the system does not reboot after an update, we find out that the backout plan is to call Dell and say "the system, it is not being working. What are we to be doing?"
  ...and we pay money for this.
  
  --
  Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
4. Re:Change management fail by TWX · 2014-07-31 11:19 · Score: 1
  
  More like same as trying a test server, then after that appears to work, pushing out to only a portion of your production cluster, if the backend can handle having two different versions running concurrently, then seeing if those portion that are changed continue to work at least as well as the original ones for a functional duration, if not better. Then push out to the rest.
  
  Even better, if the circumstances require significant DB changes, you back up the DB (you are backing up the DB, right?), start bringing up new nodes, add those nodes to the load balancer, then start bringing down old nodes. You also date-stamp the changelog so that if you have to rollback to a previous database for previous-generation servers, you can propagate the necessary changes that were committed to the new database post-roll to the old one, if you have to bring the old one back online.
  
  --
  Do not look into laser with remaining eye.
5. Re:Change management fail by Anonymous Coward · 2014-07-31 11:33 · Score: 4, Insightful
  
  Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no". The answer is always "yes" or some other affirmative that makes you think they have it under control and can do the work. When the fact is, the work they just said "yes" to, they don't actually have a clue how to perform it, so they learn as they go, on your production servers. They don't know what development / test environments are.
6. Re:Change management fail by ruir · 2014-07-31 11:42 · Score: 1
  
  Forget the obvious rollback plans? At the very least working without computers, no? Or having alternative system for something so crucial...If computers are only there to complicate, than we are better off without them. The level of incompetence and lack of planning is strong in several levels.
7. Re: Change management fail by ruir · 2014-07-31 11:51 · Score: 1
  
  DevOps also have maintenance, and production, pre-production and development systems. Automating things does mean throwing standard procedures of the industry out of the window, it only means working smarter. ...if you want to troll discussions and denigrate people who apparently work better than yourself, at least learn how to pretend that you know what you are talking about
8. Re:Change management fail by roc97007 · 2014-07-31 12:11 · Score: 3, Informative
  
  Oh man, don't get me started. It's not even clear that one would need to pay more -- we have not saved money so far by outsourcing, although the outsource company keeps telling us that savings are just around the corner. The first year, the excuse was that there is always startup issues, the second year, the excuse was that the outgoing employees did not document their jobs well enough, (probably true -- who would?) the third year the excuse was that the scope was bigger than we said it was. And so forth. Each year a new excuse and each year the total cost is more than what we were paying when we had our own IT department.
  So yeah, insourcing, or at least selective insourcing, (let them keep doing what they do well, if anything) makes tremendous sense to me.
  But I don't make the decisions.
  And even where upper management has considered terminating our outsourcing contracts, it's only to give the contract to a different outsourcing company, which only means we're now calling a building across the street from the original building in Hyderabad. Who knows, we might even be dealing with some of the same people.
  
  --
  Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
9. Re:Change management fail by dave562 · 2014-07-31 12:14 · Score: 1
  
  You bring up a good point. Given the extent out of the impact of the change, they probably should have just declared a disaster and gone with Plan B. Yet, given that they blew a system change and did not have a rollback plan, I am fairly confident that any sort of DR strategy is equally broken and worthless.
  Situations like this always put a smile on my face, because I know that my job is secure. If an organization as large as the United States government cannot get these basics right, but I can... I know that I will always be in a position to make improvements somewhere, and will never be faced with a shortage of things to do.
  I see the same thing with the major Fortune 50 corporations that I work with. I am thoroughly convinced that from the smallest shops, up to the largest organizations, the majority of IT departments are barely functioning and are just one bad change away from serious down time with no hope of recovering in any sort of reasonable amount of time.
10. Re:Change management fail by ruir · 2014-07-31 12:28 · Score: 1
  
  This kind of situation is unthinkable in 2014...haven't they heard about virtualisation technologies and clones before major changes (as at least a very minimum baseline?) How about pre-production system to TEST changes? Change management process? Backups? Alternative systems for disasters replicated from last week systems? If I had to guess, I would say they where hacked and could not count on backups, but then we are dealing with the government and DHS, and they are often known for not being that competent, everything is possible. I also agree we wont be anytime out of work. The only thing that worries me is that no matter how things are fucked up, and no matter how you are seasoned on the industry, often people want people to work for peanuts.
11. Re:Change management fail by ruir · 2014-07-31 12:33 · Score: 1, Insightful
  
  Outgoing employees are a flaw THEIR lack of management and a flaw of their processes, and I would not even accept to hear such (lame) excuse. And if there are no savings, I would bring the IT back, at least you can make them accountable and control the process better. I have worked with Indian people in the past, or rather had competing businesses employing them, and was not really that impressed with their technical abilities. And then, it probably does not help most of those firms tend to work on the cheap. Low salaries policy dont attract the best and the brightest.
12. Re:Change management fail by ShanghaiBill · 2014-07-31 12:38 · Score: 4, Informative
  
  Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".
  It takes two to fail to communicate. You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.
13. Re:Change management fail by dave562 · 2014-07-31 12:41 · Score: 3, Interesting
  
  As much as I am not a fan of government regulation, my professional experience has shown me that the only time people get IT anywhere close to right is when there is a risk of financial penalty involved in getting it wrong. Regulation seems to be the only solution to people working for peanuts. The people who work for peanuts make mistakes. If those mistakes cost the company more than the company saves by hiring those people, they will not hire those people.
  Out of all of the industries that I have worked with, the financial services industries seem to be the most together. They are not perfect, but the penalties associated with losing customer data makes them more careful.
14. Re:Change management fail by Anonymous Coward · 2014-07-31 12:44 · Score: 2, Insightful
  
  You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.
  Do I care?
15. Re: Change management fail by plopez · 2014-07-31 12:48 · Score: 1
  
  From my experience with developers, developers as Ops are a misshapen twisted wandering shambling nightmare.
  
  --
  putting the 'B' in LGBTQ+
16. Re:Change management fail by Anonymous Coward · 2014-07-31 12:50 · Score: 1
  
  No dummy. Passports are issued by the State Department. Totally not DHS.
17. Re:Change management fail by pla · 2014-07-31 12:59 · Score: 1, Insightful
  
  It takes two to fail to communicate. You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.
  
  Sorry, what part of paying you to do a job requires me to give a shit about whether or not your failed third-world culture doesn't like answering direct fucking questions?
  
  "Rude" does not apply. Breach of contract, however, does. I just wish more companies would catch on to this before they decide to outsource, rather than paying extra for literally nothing more than a built-in scapegoat for any and all problems.
18. Re:Change management fail by roc97007 · 2014-07-31 13:02 · Score: 2
  
  Exactly. Consider the business model -- A company tells you that you can outsource your IT department, buy it as a service, and pay the outsourcing company's overhead and significant profit margin, *and* save money. And the only way this could possibly work is if the outsourcing company goes to the LCC (least cost country) and hires the cheapest labor possible. This is justified in that all that IT stuff, it's all just following procedures, and anyone can do that.
  And of course, this is a blatant falsehood, but executives of the victim company either (a) don't know that, (b) *want* to believe the scam, (like any good scam) or (c) don't care, because they intend to take their bonus and get the hell out of dodge.
  So, outsourcing companies go into contracts *knowing* that cutover is going to be a Big Fail, and they have excuses prepared for when it happens. And a strategy (a brilliant strategy, really, executed by brilliant if unethical people) to string the con along as long as possible.
  And just incidentally, the victim's attempts to train the workers that they are stuck with also fails in the long run, due to the nature of the business model. The business model breaks down if the workers are paid more than starvation wages, and workers with a bit of experience can easily get a higher paid job elsewhere (perhaps as a second or third level admin for a different outsourcing company) and they quit. And then you have a new person who doesn't know what a kernel is, and you have to train them up. And all you're doing is giving out free training. How noble of you.
  ...so not only is the system *designed* to fail, the system is very specifically designed to fail continuously.
  But at least it's cheap. Oh, wait...
  
  --
  Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
19. Re:Change management fail by gstoddart · 2014-07-31 13:07 · Score: 1
  
  Yeah I was going to say the same thing. You NEVER make a change that you don't have a way to backout to the previous operational state.
  And, really, if you have something which Absolutely Has To Be There ... you make damned sure you have an environment you apply the changes to first. So that you can apply the changes and at least try to make sure stuff don't break without messing up the real one.
  This is basic change management.
  (And, yes, I am saying this without any context for this outage -- but, really, if you maintain a production environment for critical software, this is what you do)
  
  --
  Lost at C:>. Found at C.
20. Re: Change management fail by pla · 2014-07-31 13:13 · Score: 1
  
  Sorry but DevOps requires you upgrade all servers at the same time very fast, with no regard to individual server ordering.
  
  Did you mean NetOps? DevOps refers to a development paradigm. If your development paradigm risks actual user-impacting down-time, you need firing ASAP.
  Assuming you meant NetOps, can they live with provisioning me at least four (dev, test, UAT, and training) clones of the entire production environment? No? well then, they can make their case to the CTO whether inconveniencing them or our end users will have more of an impact on the bottom line. If the CTO says "go", hell, I'll code right in the production environment - Oh, you wanted that mortgage payment to go through this week? Bummer!
  
  Developers should never have the power to affect end users. If they do, it represents a failure not on their part, but on the entire IT corporate food chain, all the way to the top. Choosing customer-facing downtime over a few more terabytes and VMs amounts to corporate suicide.
21. Re:Change management fail by ShanghaiBill · 2014-07-31 13:25 · Score: 5, Insightful
  
  Sorry, what part of paying you to do a job requires me to give a shit about whether or not your failed third-world culture doesn't like answering direct fucking questions?
  
  The part about you paying them far less than you would pay someone culturally compatible. If you want to pay peanuts, you need to deal with the cultural consequences. I have dealt with Indians for years, and have learned how to ask questions so that I get the answer I am looking for. It is not that hard.
22. Re:Change management fail by ruir · 2014-07-31 13:38 · Score: 1
  
  I often can not get the mentality of wanting to have good and competent resources for peanuts. People work for a living, if customers pay, at least pay too to the people that helps running your business along. And you have summed quite nicely why I avoid working for shops whose core business is not IT, IT are viewed like some strange pumbling guys that just suck money (i.e a center cost). And then, they probably pay more to the proper plumbers...
23. Re:Change management fail by Anonymous Coward · 2014-07-31 13:42 · Score: 1
  
  That must have been the most self- and Western-centric comment I've read in a while. It wouldn't hurt to realize that next to the cultural sensitivity towards direct questions, the concept of a "contract" is also wholly different in the East than in the West...
  (Coming from a Westerner living and working in China for the past years)
  Captcha: "shudders"...
24. Re:Change management fail by Ol+Olsoc · 2014-07-31 14:15 · Score: 1
  
  Rollback plan? What is that?
  Umm - have you tried rebooting the computer sir. Try that first.
  
  --
  The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
25. Re:Change management fail by Areyoukiddingme · 2014-07-31 14:16 · Score: 4, Funny
  
  Well there's your problem! God has no part in an IT management plan.
  Yeah, the other guy has it well in hand.
26. Re:Change management fail by Ol+Olsoc · 2014-07-31 14:25 · Score: 1
  
  That must have been the most self- and Western-centric comment I've read in a while. It wouldn't hurt to realize that next to the cultural sensitivity towards direct questions,
  I't also the silliest fucking thing I've ever heard. If I have the director and a couple higly placed people breathing down my neck at a phenomenal burn rate, I gotta say that if the dude from India cannot answer simple yes or no questions, The's the last of their computers that will ever enter the shipping dock.
  They're giving support to Americans. This booboo feelings sensitivity BS is just one more reason the offshoring doesn't work. If I'm insulting the poor guy, I wont ever bother him or his company again.
  
  --
  The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
27. Re:Change management fail by roc97007 · 2014-07-31 14:26 · Score: 3, Funny
  
  I sometimes think that if I accidentally entered a church with an IT management plan in a back pocket, my pants would burst into flames.
  
  --
  Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
28. Re:Change management fail by l0n3s0m3phr34k · 2014-07-31 14:27 · Score: 3, Interesting
  
  This isn't always the case. A company can save money via outsourcing IT infrustructure if they go with the right vendor. VMWare, virtual servers, proper fail-overs, big multi-core blade racks were the VM is still more powerful than your original server and still costs less...but of course I work at HP in the Enterprise Services so the level I'm talking about probably isn't affordable for a "small company". We have VERY specific steps for everything, our "runbooks" detail everything from server configs, hardware, rack enclosures, port layouts, and responsible parties to contact for each part if it fails. When you have a rack of blades, it's far easier to snapshot, launch then test,,,we always have a "backup" in a hot image ready to go if anything fails. Often I'm working with 5-15 people spread across the globe all doing different functions (unix admin, wintel, recovery, netops, etc) but we rarely have any "HP owned" customer-impacting outages. Of course my major clients are airlines so it's all tightly regulated; your individual milage may vary LOL.
29. Re:Change management fail by Ol+Olsoc · 2014-07-31 14:28 · Score: 1
  
  I often can not get the mentality of wanting to have good and competent resources for peanuts. People work for a living, if customers pay, at least pay too to the people that helps running your business along. And you have summed quite nicely why I avoid working for shops whose core business is not IT, IT are viewed like some strange pumbling guys that just suck money (i.e a center cost). And then, they probably pay more to the proper plumbers...
  Don't worry, the cloud's basic purpose is to get rid of those IT guys.
  
  --
  The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
30. Re: Change management fail by l0n3s0m3phr34k · 2014-07-31 14:31 · Score: 1
  
  Sounds like you have an Ghisguth infection. I suggest calling Dresden the Wizard and maybe he can help out!
31. Re:Change management fail by ruir · 2014-07-31 15:07 · Score: 1
  
  Sure, the cloud runs itself magically, and you call Merlin when you have problems...That should be the most stupid comment I have read lately. The cloud is only outsourcing, you know? It is like you saying you dont need more mechanics because you go to the garage and leave there your car. ;-P
32. Re:Change management fail by Anonymous Coward · 2014-07-31 15:08 · Score: 1
  Yeah I was going to say the same thing. You NEVER make a change that you don't have a way to backout to the previous operational state.
  Sounds great to MBA types, but easier said than done, the real world don't have a transaction log or a time machine you can just rollback. Especially when management decreed that ANY action not regularly scheduled automatically is considered a "change".
  Tell me, how do you "backout" of these "changes" in a way you can be SURE you would be back to the "previous operational state":
  - copying a file from directory A to directory B (how can you be 100% SURE your "backout" job would delete to correct file?)
  - restoring a database from backup
  - reorganizing a database
  - rebuilding an index
  - taking a backup
  - collecting and updating the statistics of your database tables
  - shutting down or rebooting a machine without any configuration change (what if it won't start? How do you "backout"?)
  - shutting down or restarting a process on the machine without any configuration change (what if it won't start? How do you "backout"?)
  - plugging a network cable to another socket (double penalty for answering "plugging it back to the original socket", it doesn't guarantee operational state if the cable plug lost a pin in process)
  - replacing any hardware component ("putting the old one back in" in not an acceptable answer, see above)
33. Re:Change management fail by khchung · 2014-07-31 15:13 · Score: 4, Insightful
  
  Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".
  On the other hand, I have encountered plenty of managers who literally do not know how to take "no" as an answer.
  Takes two to make a pair.
  
  --
  Oliver.
34. Re:Change management fail by Ol+Olsoc · 2014-07-31 15:24 · Score: 1
  
  Sure, the cloud runs itself magically, and you call Merlin when you have problems...That should be the most stupid comment I have read lately. The cloud is only outsourcing, you know? It is like you saying you dont need more mechanics because you go to the garage and leave there your car. ;-P
  You don't think the suits are salivating over the possibilitiy of the people they can get rid of once they send as much work as possible to "the cloud"? There is a repeating cycle in the business world. They start out with in-house. Then they want to cut costs, so they outsource, then outsourceing doesn't work very well because of either costing more than it was supposed to, or it doesn't work like it was supposed to, so they start hiring in house again(and remember, you are just another customer to your cloud supplier, so you don't get any special treatment - same as outsourcing) Then they run with in house for a while, then they want to cut costs.....
  I've seen the cycle many times.
  
  --
  The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
35. Re:Change management fail by Anonymous Coward · 2014-07-31 15:50 · Score: 2, Insightful
  
  You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.
  Do I care?
  If affects your ability to do your job, you should.
36. Re:Change management fail by JustOK · 2014-07-31 15:54 · Score: 2, Insightful
  
  Why? Are you a liar?
  
  --
  rewriting history since 2109
37. Re:Change management fail by JustOK · 2014-07-31 16:07 · Score: 1
  
  Sounds like your feelings are hurt.
  
  --
  rewriting history since 2109
38. Re:Change management fail by roc97007 · 2014-07-31 16:14 · Score: 1
  
  Sorry, I guess I deserved that. I meant, of course, that the plan would burst into flames.
  
  --
  Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
39. Re:Change management fail by roc97007 · 2014-07-31 16:16 · Score: 1
  
  Not to put too fine a point on it, but I've heard that sales pitch before. I agree, HP has some truly brilliant admins, as evidenced by the "demo". The day to day reality is very much different.
  
  --
  Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
40. Re:Change management fail by Ol+Olsoc · 2014-07-31 16:45 · Score: 1
  
  Sounds like your feelings are hurt.
  What an odd reaction. I did what I did and was paid very well for it because I got things done. If the help doesn't perform, you simply change the help.
  
  --
  The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
41. Re:Change management fail by davester666 · 2014-07-31 18:11 · Score: 1
  
  Sure the state department may issue them, but DHS controls the database. It's one of the primary data sources for their terrorist watch list database.
  
  --
  Sleep your way to a whiter smile...date a dentist!
42. Re:Change management fail by bayankaran · 2014-07-31 18:51 · Score: 2
  
  Outsourcing is bad, but as an Indian who renewed his passport recently, I have to say the process was smooth.
  The document verification/IT/software/hardware part of the Indian passport application/renewal operation is now handled by TCS. You make an appointment online - for both fast track and slow track - and arrive at the local passport center with your documents. The TCS grunts allot a token, make sure documents are in order - if they are not you are sent back to get them corrected, take your fingerprints, photo and for final verification you meet an Indian government employee. He/she either says YES, or says YES with police verification.
  In my case it was the latter as my passport had some damage - usual wear and tear, but visible.
  A few days later a policewoman came home after fixing an appointment over phone, verified my documents/address and in two weeks I got a fresh passport.
  I was impressed by the whole process. AFAIK Indian government and TCS is doing everything right as far as passport renewal goes...it was better than getting my experience in getting a drivers license/immigration papers in Toronto/Chicago/San Francisco.
  May be US government should indeed outsource the operation to TCS...if they can do a good show in an anarchic chaotic mess like India, I am sure they can do the same in US.
  
  --
  Tat Tvam Asi
43. Re:Change management fail by dbIII · 2014-07-31 19:01 · Score: 1
  
  There are sometimes very "political" problems that prevent rollback because for instance that means buying more time on the licences from an earlier vendor when the new vendor is heavily embedded with the management that are driving the change. Such a problem in my state resulted in such an enormous fuckup in Hospital payroll systems with no rollback that there was real political fallout - after the bill went beyond 500 million the government lost office to be replaced by a bunch of baby fascists led by a Pocket Putin.
44. Re:Change management fail by Stolpskott · 2014-07-31 19:25 · Score: 1
  
  ...in case my other article did not make it clear, we always ask if they have a backout plan, and they always say they do.
  I used to deal with a lot of Indian outsourced IT groups, and the only way to handle this is to either follow up the "Yes, we have a backout plan" response with "Tell me what your backout plan is" or just to skip straight to that without bothering to ask the "Do you have a plan?" question.
  Things still got screwed up, but after the first occurrence we completely cut their access to the servers and re-enabled them on demand, so we forced their people to update a specific server first to show that they could do it on a system which is not mission-critical.
  However, that approach really only works when the client does not turn into a whining tub of lard when the vendor starts putting pressure on.
45. Re:Change management fail by ShanghaiBill · 2014-07-31 19:28 · Score: 5, Interesting
  
  simple yes or no questions
  It is only simple because you speak English. You need to widen your cultural perspectives. In other languages, and other cultures, it is not so simple. For instance, Chinese does not even have the words "yes" and "no". If you ask a Chinese speaker if they have a pen, they will answer "have" or "not have". If you ask them if they are going to lunch, they will answer "going" or "not going". There is no such thing as a "yes or no question" in Chinese, and culturally, Chinese are much more direct than Indians or Japanese.
46. Re:Change management fail by ruir · 2014-07-31 19:52 · Score: 1
  
  Very interesting insight sir, we often are very aware of the political problems in our backyard, but forget others have such similar problems.
47. Re:Change management fail by hawkinspeter · 2014-07-31 19:54 · Score: 2
  
  All of those are trivial to rollback. Just create a snapshot of the VM before the change and you can revert back to it if there's any problem.
  
  --
  You're a temporary arrangement of matter sliding towards oblivion in a cold, uncaring universe
48. Re:Change management fail by ruir · 2014-07-31 20:00 · Score: 5, Insightful
  
  Well, it also affects my ability to do the job people lying to me or choosing to reply with half truths to save face. My culture considers that extremely rude too. The rules of engagement have to change in a multicultural world, and if I am the customer, their obligation to bend somewhat their culture is a ball on their side. Or I may take my business and wallet elsewhere.
49. Re:Change management fail by ruir · 2014-07-31 20:02 · Score: 2
  
  The question is that exactly because of that culture, they seem able to complicate the most simple of the problems looking through the eyes of our culture. It also does no help that they say they know everything, their CV looks like far better than a Linus Torvalds, lots of IT certifications, but when your start asking them the simplest of the questions, they fall apart.
50. Re:Change management fail by Loki_1929 · 2014-07-31 21:00 · Score: 1
  
  Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".
  It takes two to fail to communicate. You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.
  So they lie because their culture tells them to and it's my fault for not identifying that they're lying and taking careful steps to help them not lie?
  Sorry, but that's absurd. If one's culture does not allow one to perform one's job correctly, one needs to either find a new culture or find a new job.
  
  --
  -- "Government is the great fiction through which everybody endeavors to live at the expense of everybody else."
51. Re:Change management fail by Jesrad · 2014-07-31 21:05 · Score: 3, Insightful
  
  Sounds like your IT has been outsourced to India
  Not necessarily. I've seen this exact kind of madness happen just as easily with locals, here in France. Like that time the local, on-site support team from our vendor rebooted the production server instead of the test platform, because woops wrong terminal window in the foreground.
  Or when they covertly rolled out a "shame-bug fix" remotely on the production platform during a week-end night, again instead of targetting the test platform, then noticed their mistake, and wiped-out months of production data by reverting to a long-expired backup.
  Or when the local datacenter people managed to botch our fully-automatized install+deploy+configure solution by messing up on the one thing they had to do right - that is, upload it and launch it on the correct machine of the cluster.
  Don't think hiring local people for more money protects you from such cringe-worthy nonsense. The moment you outsource anything, and I do mean *anything*, no matter how far and how expensive and what nationality: if you base your expectations on anything but an actual track-record of reliability and dependability, you're exposing yourself to long hours of hair-pulling and yelling into phones.
  
  --
  Maybe we deserve this world ?
52. Re:Change management fail by Dr.+Gamera · 2014-08-01 00:47 · Score: 2
  
  Appreciate your comment. Can you provide some examples of how you would word questions to get a useful answer?
53. Re:Change management fail by arglebargle_xiv · 2014-08-01 01:02 · Score: 1
  
  I have dealt with Indians for years, and have learned how to ask questions so that I get the answer I am looking for. It is not that hard.
  Paleface no ask question with forked tongue, paleface get correct answer.
54. Re:Change management fail by dave420 · 2014-08-01 01:34 · Score: 2
  
  You use a second machine, doofus. It's not rocket science. Mirror A to B, make changes to B, switch to B, if B breaks, switch back to A, if B doesn't break, you're golden.
55. Re:Change management fail by Ol+Olsoc · 2014-08-01 01:45 · Score: 1
  
  simple yes or no questions
  It is only simple because you speak English. You need to widen your cultural perspectives. In other languages, and other cultures, it is not so simple. For instance, Chinese does not even have the words "yes" and "no".
  I worked with a Gentleman from China. He said yes and no it with the ugliest of us Americans he worked with.
  And hey, if I was working in China, dealing with culture, and and saying yes or no was offensive - then I would do as they do.
  But I'm here in America, where we do talk that way. If I have to learn the social quirks of every country I'm dealing with, along with not being able to understand the "help" half of the time, it's not going to happen. The source of help is a big part of the buying decision.
  You are only one small logical step away from saying that the ugly American has to learn the language of anyone he's dealing with in support.
  
  --
  The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
56. Re: Change management fail by Ol+Olsoc · 2014-08-01 01:50 · Score: 1
  
  At least he not a coward and is using his alias. People may fall in love with a place, and at least he is trying to improve his life abroad than being exploited back home...doing more than you probably.
  
  Why would he be exploited back home?
  This America you hate - I do not think it is as you think it is.
  People think we're all kooks and nuts because we allow the kooks and nuts to have their say.
  
  --
  The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
57. Re: Change management fail by operagost · 2014-08-01 02:24 · Score: 1
  
  Having excellent language skills does not mean you are intellectually superior any more than my being able to play many musical instruments does.
  
  --
  
  Gamingmuseum.com: Give your 3D accelerator a rest.
58. Re:Change management fail by ruir · 2014-08-01 02:47 · Score: 1
  
  Well, the feelings are pretty much irrelevant on both sides. Business is business, cognac is cognac.
59. Re: Change management fail by ruir · 2014-08-01 02:51 · Score: 1
  
  Are you the asshat of the jour today? I am not american btw, and I was giving a tongue on cheek answer to the idiocy of the anon coward.
60. Re: Change management fail by ruir · 2014-08-01 02:53 · Score: 1
  
  If you say so. I worked 6 years of my life in a backwards country, but I managed to earn and save enough to have today a slightly more comfortable life.
61. Re:Change management fail by operagost · 2014-08-01 02:55 · Score: 1
  
  Of course, someone who is learning Mandarin needs to understand that. But someone who is Chinese needs to understand the concept of "yes" and "no" in English. Why hold them to a lower standard-- are they subhuman?
  
  --
  
  Gamingmuseum.com: Give your 3D accelerator a rest.
62. Re: Change management fail by Ol+Olsoc · 2014-08-01 02:59 · Score: 1
  
  Godamned right I'm an asshat. That you are not American is really obvious, in case you don't know it.
  And that might just make both of us happy.
  Because at least you don't have a fatso at home.
  
  --
  The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
63. Re:Change management fail by phorm · 2014-08-01 03:00 · Score: 2
  
  Yes=en. Its a short syllable pronounced from the throat with the mouth closed (somewhat grunt-like but not so inelegant). This is effective for most interrogatives.
  No is roughly "bu" (boo). This is more a negative so it is often combine with other words, but "bu xie" (Boo shie / no thank you) is generally effective, but in the case of the "do you have" question a straight "boo" wouldn't be correct and the "I don't have" variety is useful. Conversely, "do you want [my pen/to go for lunch]" is generally answerable with "bu xie"
  Maybe the parent should swap "simple positive and negative" instead of "yes and no", but the intent is the same and quite often it's not a language issue are you've put forth so much as a culture issue in terms of providing a straightforward response (whether it be ethnic culture or industry culture, etc).
  It quite often still comes down to certain people or groups (companies etc) just being naturally evasive towards a straightforward answer.
64. Re:Change management fail by JimFive · 2014-08-01 04:06 · Score: 1
  
  How about, instead of asking "Do you have an action plan?" ask, "What is your action plan?" or "Please send me your action plan."
  
  --
  Please stop using the word theory when you mean hypothesis.
65. Re:Change management fail by Hans+Adler · 2014-08-01 04:28 · Score: 1
  
  I don't think it's so hard. Extreme example:
  "Some of our computers are more important for our business than the others. For example, every hour that kremvax is not running costs us roughly half a million dollars because people worldwide then buy somewhere else.
  It often happens that new firmware from the vendor does not work properly. So suppose you make a firmware update on kremvax, reboot it, and it does not start. What will you do in that case?" ...
  "That's probably the best thing you can do in this situation, though I would ask you to also ... so that ... . Unfortunately, if this happens, it can take days to resolve the situation. Two days at half a million dollars each, that would make, um, ... How much is that, help me out please..." ...
  "Yes, exactly, 24 million. That's why our last system administrator was always very scared. He wouldn't have lost his job if this happened once, but some people would have been very angry. So he was always reading those books about anticipatory system administration [I just made this term up; use a proper, searchable technical term if there is one] to minimize the danger. He always came up with new tricks. For example, there is this server called deepthought that is not doing anything really important most of the time. We call this a development server. Now when he wanted to do a firmwareupdate, what do you think: which of the servers did he do first, kremvax or deepthought?"
66. Re:Change management fail by Hans+Adler · 2014-08-01 04:43 · Score: 1
  
  The problem is that that's often not how they think. In some cultures you only ask a yes or no question in order to hear a yes answer, no matter what. You think you are enquiring for information. The poor Elbonian you are talking to may be totally stressed out because he thinks you are telling them that you are very angry about him because you lost face before your superior due to his poor performance. And that, even though nobody knows or can know how to do things better, you now have to report back to that superior, untruthfully (as everybody including the superior knows), that the Elbonian will solve the problem. You just need a yes from the Elbonian, so it can travel up the chain of command. The Elbonian just needs to confirm that he is doing that impossible thing that nobody has ever heard of, and everything will be fine for a while.
  If, instead, you wanted him to try to actually do this, then you would - in the mind of the Elbonian - ask very differently.
  Now of course you can try to find Elbonians who know how to communicate with Americans. Just don't be surprised if it's hard, and maybe impossible at the price you are willing to pay.
67. Re:Change management fail by roc97007 · 2014-08-01 05:02 · Score: 1
  
  Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".
  It takes two to fail to communicate. You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.
  Um, ok.... this is IT. Sometimes yes or no questions are required. If the outsourced IT company can't culturally or by whatever reason answer a yes or no question, should we be doing business with them?
  I mean, if this is a cultural problem throughout India (and I don't really believe that's true) how do they have, for instance, a space program? How can anything that requires precise communication be made to work?
  Is it possible that the real reason why they always answer "yes" even when they clearly don't understand the question (we've seen many many examples of this) because we are dealing with people who have been hired to do a job that they are unqualifed to do but are required to keep up appearances with the customer?
  
  --
  Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
68. Re:Change management fail by roc97007 · 2014-08-01 05:13 · Score: 1
  
  The question is that exactly because of that culture, they seem able to complicate the most simple of the problems looking through the eyes of our culture. It also does no help that they say they know everything, their CV looks like far better than a Linus Torvalds, lots of IT certifications, but when your start asking them the simplest of the questions, they fall apart.
  Given that there's only about a billion of them, I would say it is safe to assume that they all fall apart and that there isn't a single Indian how actually knows anything.
  It's almost a miracle that they've managed to come this far.
  That's a bit disingenuous. We're talking specifically about the people hired by companies in the business of outsourcing IT. That has to be a pretty small subset of a billion.
  And as I said in another article, it's not that way because some entire race of people are stupid, but because of the way the business model is set up. In order to offer IT labor at an attractive price while including overhead and profit margin, it becomes necessary to buy the cheapest labor possible. This tends to present to the customer workers that are, shall we say, less than capable. And inflated resumes seems to be a real thing.
  
  --
  Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
69. Re:Change management fail by roc97007 · 2014-08-01 05:20 · Score: 1
  
  I don't think it's so hard. Extreme example:
  "Some of our computers are more important for our business than the others. For example, every hour that kremvax is not running costs us roughly half a million dollars because people worldwide then buy somewhere else.
  It often happens that new firmware from the vendor does not work properly. So suppose you make a firmware update on kremvax, reboot it, and it does not start. What will you do in that case?" ...
  "That's probably the best thing you can do in this situation, though I would ask you to also ... so that ... . Unfortunately, if this happens, it can take days to resolve the situation. Two days at half a million dollars each, that would make, um, ... How much is that, help me out please..." ...
  "Yes, exactly, 24 million. That's why our last system administrator was always very scared. He wouldn't have lost his job if this happened once, but some people would have been very angry. So he was always reading those books about anticipatory system administration [I just made this term up; use a proper, searchable technical term if there is one] to minimize the danger. He always came up with new tricks. For example, there is this server called deepthought that is not doing anything really important most of the time. We call this a development server. Now when he wanted to do a firmwareupdate, what do you think: which of the servers did he do first, kremvax or deepthought?"
  And you realize, by doing this you are training up a very very junior person, because experienced sysadmins should already know all of this.
  And this is the point. We are told that we are paying for experienced sysadmins. What we get are people who need to be taught that you do a firmware upgrade on a test system first before you do it in production.
  Incidentally, once you get that admin trained up and things running smoothly, he will find a job elsewhere for more money and you will have to start over. Because for the business model to work, first line admins have to be paid starvation wages.
  This is not an ethnic cultural issue. It's a business cultural issue.
  
  --
  Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
70. Re:Change management fail by roc97007 · 2014-08-01 05:24 · Score: 1
  
  simple yes or no questions
  It is only simple because you speak English. You need to widen your cultural perspectives. In other languages, and other cultures, it is not so simple. For instance, Chinese does not even have the words "yes" and "no". If you ask a Chinese speaker if they have a pen, they will answer "have" or "not have". If you ask them if they are going to lunch, they will answer "going" or "not going". There is no such thing as a "yes or no question" in Chinese, and culturally, Chinese are much more direct than Indians or Japanese.
  I think you're playing with words. The original stipulation, to which many of us are objecting, is the inability to answer a direct question in a direct manner. If you ask a Chinese speaker if he has a backout plan, and he answers "have" and actually means it, and isn't just faking it because he doesn't understand the question then he has answered the question. Language quirks are irrelevant.
  
  --
  Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
71. Re:Change management fail by roc97007 · 2014-08-01 05:34 · Score: 1
  
  I think cutting their access and adding it back on a case by case basis is a brilliant idea. But I fear we as a company have lost control of the environment. We don't have root anymore, except for the service accounts, and sudo lists that they've forgotten to, or don't know how to, plug. And we can't seem to convince upper management that even the developers who used to be admins could do a better job than the people we have now.
  
  --
  Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
72. Re:Change management fail by BobMcD · 2014-08-01 07:10 · Score: 1
  
  I do find it interesting how the onus is on the native to adapt to the foreigner ONLY in your outsourcing situation.
  Or would you likewise expect Chinese natives buying services from an American company to adapt?
73. Re:Change management fail by ihtoit · 2014-08-01 08:25 · Score: 1
  
  DHS doesn't control the database, they outsourced it to Serco who outsourced it to G4S who outsourced it to a call centre in Bangalore who outsourced it to IBM.
  
  --
  Political debates have me rolling my eyes so much I think I got optical whiplash. I should sue. - Foamy The Squirrel
74. Re:Change management fail by ihtoit · 2014-08-01 08:29 · Score: 1
  
  This. So THIS! I cannot and will not do business with liars. If I can't get a straight answer to a straightforward question, I'll go elsewhere.
  It also helps my clients if they're up front, honest and straightforward with me. In ascertaining their requirements, I ask exacting questions, if they think I'm being anal, well, they just found out why I'm so fucking expensive. It's because the job isn't done until it's right.
  
  --
  Political debates have me rolling my eyes so much I think I got optical whiplash. I should sue. - Foamy The Squirrel
75. Re:Change management fail by ihtoit · 2014-08-01 08:31 · Score: 1
  
  there's a viable industry around inflated resumes. I've seen some real gems.
  
  --
  Political debates have me rolling my eyes so much I think I got optical whiplash. I should sue. - Foamy The Squirrel
76. Re:Change management fail by wwphx · 2014-08-01 09:18 · Score: 1
  
  ...The backout plan is to engage the vendor.
  Preferably with shells from an Iowa-class battleship?
  
  --
  When you sympathize with stupidity, you start thinking like an idiot.
77. Re:Change management fail by ruir · 2014-08-01 14:56 · Score: 1
  
  I often have this problem with suppliers and in the past even had it with subordinates. Nowadays no suppliers at all get root, and I have configurations in place where all their commands are logged to the system logs.
78. Re:Change management fail by PJ6 · 2014-08-02 06:01 · Score: 1
  
  Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".
  It takes two to fail to communicate. You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.
  Really? Could a question like, "Is it raining?", which presumably would be answered by a yes or a no, really be considered rude?
  
  I'm at a total loss to see how any interpretation of your assertion could be true.
  
  Could you give us a simple and clear example, and then provide another in the context of a work environment, where management is expected and acceptable?
Maybe the IRS can help by Bodhammer · 2014-07-31 11:00 · Score: 1

They are hard drive experts!

--
"I say we take off, nuke the site from orbit. It's the only way to be sure."
Damn by rmdingler · 2014-07-31 11:05 · Score: 1

Still, bet Sysadmin's the highest ranking head that'll roll.

--
Happiness in intelligent people is the rarest thing I know.
Ernest Hemingway
1. Re:Damn by ruir · 2014-07-31 11:52 · Score: 1
  
  Will they? Apparently they do not have sysadmins and are just a bunch of system operators, if that.
The solution by Loopy · 2014-07-31 11:11 · Score: 4, Funny

Sic the healthcare.gov guys on it. I'm sure it'll be right as rain in no time.
Replication != Backup by Anonymous Coward · 2014-07-31 11:19 · Score: 5, Interesting

From their Q&A:
Q: Why wasn’t there a back-up server?
Back-up capability and redundancy are built into the system. The upgrade affected our current processing capability, in part because it interfered with the smooth interoperability of redundant nodes.
We don't need backups, the data is replicated, we're cool.
1. Re:Replication != Backup by ruir · 2014-07-31 11:53 · Score: 1
  
  this one smells like BS even very far, far away. I change contractors on the spot.
2. Re:Replication != Backup by Noah+Haders · 2014-07-31 11:59 · Score: 1
  
  I would change governments on the spot. Kiss my ass, ins/dhs/TSA!
3. Re:Replication != Backup by sumdumass · 2014-07-31 12:01 · Score: 4, Funny
  
  http://xkcd.com/327/
  I wonder if little bobby tables applied for a passport while this was going on?
Over Improved by bfmorgan · 2014-07-31 11:27 · Score: 1

We call this being over improved. So much for testing.

--
I hope this caused some synapses to fire.
One Database to rule them all by sehlat · 2014-07-31 11:32 · Score: 1

One Database to bind them.
One Database to keep them out.
And into the darkness send them.
Ask the NSA by Daemonik · 2014-07-31 11:33 · Score: 4, Funny

I'm sure they have full copies of all the data already.
I wonder by ruir · 2014-07-31 11:36 · Score: 2

That these breakdowns are lame excuses. If computers fails, have people forgot how to do the same process manually? It is better to halt all the flights than letting people through and risk "terrorists" flying? Are we that terrified?
1. Re:I wonder by Kittenman · 2014-07-31 15:45 · Score: 2
  
  That these breakdowns are lame excuses. If computers fails, have people forgot how to do the same process manually? It is better to halt all the flights than letting people through and risk "terrorists" flying? Are we that terrified?
  You could just ask the questions that used to get asked back in the '50s. "Do you intend to bring down or otherwise defame the US government?"
  
  Really. A UK humourist (Frank Muir?) wrote "Sole purpose of visit" on the form.
  
  --
  "The greatest lesson in life is to know that even fools are right sometimes" - Winston Churchill
2. Re:I wonder by dbIII · 2014-07-31 19:17 · Score: 1
  
  If computers fails, have people forgot how to do the same process manually?
  
  Yes.
  As an example. I've been rushed to a steel mill rolling line with a pocket calculator because the operators were not taught how to divide the number on the dial of the test machine by the cross sectional area of the rod that they had measured the diameter of - they were just told to manually enter those two numbers into the computer. By knowing how to calculate the area of a circle I was saving downtime of hundreds of thousands of dollars per hour - which is pathetic on so many levels since the operators also knew how to find the area of a circle but nobody had told them that's what the computer was doing. They had only been told that the process was to put two numbers into a computer.
  
  It's not about stupid operators/counter staff it's about relying on fragile links with no workaround when they break. There's also the fear of taking responsibility - it's seen as safer for the supervisor of all those staff to say "nothing should be done" than to work around the problem.
3. Re:I wonder by phorm · 2014-08-01 03:05 · Score: 1
  
  There's process and then there's information. If a till at the supermarket breaks, once can (theoretically) still process cash purchases by hand, but how about Debit/Credit?
  As per the article, it addresses why it was not possible to implement some manual workaround : "We cannot 'handwrite' visas because security measures prevent consular officers from printing a visa unless it is approved through our database system. Until the system is brought back to full capacity and we are able to work through the backlog, service to our customers will be below normal."
4. Re:I wonder by ihtoit · 2014-08-01 08:50 · Score: 1
  
  There's process and then there's information. If a till at the supermarket breaks, once can (theoretically) still process cash purchases by hand, but how about Debit/Credit?
  Yes, just use any one of the other 68 tills.
  
  As per the article, it addresses why it was not possible to implement some manual workaround : "We cannot 'handwrite' visas because security measures prevent consular officers from printing a visa unless it is approved through our database system. Until the system is brought back to full capacity and we are able to work through the backlog, service to our customers will be below normal."
  What the hell did they do in the 1950s??
  
  --
  Political debates have me rolling my eyes so much I think I got optical whiplash. I should sue. - Foamy The Squirrel
5. Re:I wonder by phorm · 2014-08-01 10:16 · Score: 1
  
  "What the hell did they do in the 1950s??"
  I'd imagine (and in many ways hope) that security requirements and policies have changed quite a bit since the 1950's
Not just the passports by Anonymous Coward · 2014-07-31 11:48 · Score: 2, Insightful

The whole US customs and immigration system is massively dysfunctional. Last year I flew into Minneapolis from Asia. I'd been traveling for twenty hours straight and then I got to stand in line for a full hour waiting for an immigration agent to spend ten seconds looking at my passport photo to make sure it matched my face. Even the third world airports I've been through aren't that bad. There were even empty stations without agents. How much would it have cost to add a few more agents - $100? At the time they were doing this ridiculous upgrade to the airport that must have cost millions - they were setting up all these silly little tables with ipads in the waiting areas. But somehow they couldn't manage to have enough immigration agents. It made me wonder if people in the state of Minnesota are as silly as their ariport - they did elect Michelle Bachmann to congress - so there may be quite a few of them who were dropped on their heads as babies or something.
1. Re:Not just the passports by CRCulver · 2014-07-31 13:02 · Score: 1
  
  At the time they were doing this ridiculous upgrade to the airport that must have cost millions - they were setting up all these silly little tables with ipads in the waiting areas. But somehow they couldn't manage to have enough immigration agents.
  
  Airport infrastructure is typically managed by a consortium with private or state-level involvement. US customs officials are federal employees. The two parts of the airport have entirely different funding sources.
2. Re:Not just the passports by Anonymous Coward · 2014-07-31 13:26 · Score: 1
  
  The two parts of the airport have entirely different funding sources.
  In a sane world, the airport director would phone up Michelle Bachmann, explain that it reflects poorly on the people of Minnesota for international travelers to be having a 3rd world immigration experience, that many international travelers have paid thousands of dollars for their tickets while extra immigration agents can be had for something like $10/hour, and then Michelle Bachmann would get together with her buddies in congress and they would have the situation fixed by the end of the day. But, thanks to all this nonsense about the glory of privatization and local government, Minnesota is stuck with a third world mess of an airport.
3. Re:Not just the passports by PPH · 2014-08-01 01:40 · Score: 1
  
  They have most of the lines closed so they can back people up and watch who is starting to break out in a cold sweat.
  Which is why most of the bad guys fly first/business class. Faster through the lines. Less chance of being observed.
  
  --
  Have gnu, will travel.
4. Re:Not just the passports by david_thornley · 2014-08-01 05:33 · Score: 1
  
  Customs and immigration agents are Federal employees, and airport upgrades are state and private level. I don't know if an airport can simply pay Customs to supply more agents.
  In any case, having another station open would cost a whole lot more than $100. There isn't a 24/7 pool of available customs agents who can simply show up for $50/hour, two hours minimum, when there happens to be a lot of people coming through. To man a station 24/7/365 (shouldn't that be 24/7/52.14?) takes a minimum of five people working there.
  And, yes, we tend to elect more colorful politicians. You missed Rudy Perpich and Jesse Ventura as governors (Jesse "The Governor" was actually pretty good), among others.
  
  --
  "When you have eliminated the unacceptable, whatever is left, however improbable, must be the truthiness" - Holmes
5. Re:Not just the passports by david_thornley · 2014-08-01 13:33 · Score: 1
  
  Apparently you've never had to run a payroll.
  First, you're saying 8 hours a day. However, this only matters for international flights, and they can expect to get in pretty much whenever. We want to have coverage for more than when your flight happens to come in. Think 24 hours a day. That's something like 40 or 50 additional agents. Remember that number.
  Second, people cost a lot more than you actually pay them. You have to pay FICA on top of that, there's likely to be benefits of some sort, and you need to manage these guys, hire and fire, and keep track of everything. That's going to push the cost up by probably 50%.
  Third, these guys have to do more than look at passports. They have to be able to deal with what might come up as customs officers. This involves training and supervision, and you might not be able to get good enough people at $10/hour.
  Overall, then, you're massively underestimating the costs. I'm going to estimate well over a million a year to keep these stations staffed, and I'm being conservative here.
  As far as costs go, I really doubt it's general government incompetence. I suspect it's at the Congressional and Presidential level, where irrational budget decisions are made because neither party wants to go along with what the other sees as rational. (Remember the Sequester, which actually cost more money than not sequestering?)
  
  --
  "When you have eliminated the unacceptable, whatever is left, however improbable, must be the truthiness" - Holmes
Here's the problem... by Anonymous Coward · 2014-07-31 12:27 · Score: 2, Funny

I think I found the problem, from the Department of State's own website:
"The Department of State is working with Oracle and Microsoft to implement system changes aimed at optimizing performance and addressing ongoing performance issues."
They're running Oracle on Windows.
1. Re:Here's the problem... by AJWM · 2014-07-31 15:49 · Score: 4, Funny
  
  Or worse, they're running SQLServer on Sun boxes...
  
  --
  -- Alastair
A Welcome to The Crash by Mister+Liberty · 2014-07-31 12:56 · Score: 1

I have arrived at the point where any crashes experienced by whatever State Department of whatever so called and self proclaimed Democratic Country (traitor mark here) are welcomed by me with the utmost glee. The more disruption, the more chances for a turnaround.
Mission critical failure by Anonymous Coward · 2014-07-31 13:24 · Score: 1

And how many families will be disconnected because of this?
How many jobs will be lost when people can't get back to work?
It's all nice conveniently glossing over the fact that people can't get home but they have lives to live, schedules to meet and contracts their obligated to perform under. You can't just say "Oh, sorry. You can't come back. Try later." The real world doesn't accept "Try later" as an excuse.
Large Databases? by TechyImmigrant · 2014-07-31 14:46 · Score: 4, Insightful

The article tries to wow us with the hugeness of the database, like this is a reason for the issues.
Yet the numbers quoted are not that big. Any modern PC isn't going to get too upset handling 75 million things. A real data center is going to sit there wondering what to do with the remaining 500TB of storage.
I don't doubt that there is some horrible flaw in the way the system was conceived that rendered it fragile, but whatever it is, it's nothing to do with the enormity of the problem, because it isn't very enormous.

--
I should use this sig to advertise my book ISBN-13 : 978-1501515132.
1. Re:Large Databases? by TechyImmigrant · 2014-08-01 05:32 · Score: 1
  
  Ahh, Non free software. So I guess the FlexLM server went on the blink and is holding everything else up.
  
  --
  I should use this sig to advertise my book ISBN-13 : 978-1501515132.
2. Re:Large Databases? by ihtoit · 2014-08-01 08:56 · Score: 1
  
  75 million page-sized data objects (let's call it "text") would fit comfortably on my laptop hard drive*.
  *2x2TB, I feel like I'm flapping my cock out when I'm sitting in a coffee bar playing world of tanks on it.
  
  --
  Political debates have me rolling my eyes so much I think I got optical whiplash. I should sue. - Foamy The Squirrel
I understand that the database is large... by Anonymous Coward · 2014-07-31 16:10 · Score: 1

I understand that the database is very large by any measure: 100 million records, 75 million pictures. But social security databases in most countries (or income tax databases) are at least that large (ok, likely much larger). Its a fail to have a large database really tank like this. If you need to shut the whole thing down for a day to avoid corrupt data, then shut it down. Fixing a corrupt database is much more difficult than correctly shutting a (slow) one down and then bringing it back up again.
1. Re:I understand that the database is large... by david_thornley · 2014-08-01 05:37 · Score: 1
  
  One thing I learned at an Oracle user group meeting: what you or I might think a large database is actually pretty darn small by industry standards. We discussed one where the activity pretty well maxed out write capacity on their disk drives.
  
  --
  "When you have eliminated the unacceptable, whatever is left, however improbable, must be the truthiness" - Holmes
1970s design mentality? by narnian · 2014-07-31 18:57 · Score: 1

While it doesn't always go this way, often simple things like the User Experience of a business gives an indication of the ethos behind a whole lot of the processes and systems they are using. To wit, compare the US Arrivals card that all "aliens" need to use upon arrival into the US, with the one from Australia. A clear 1970s look-and-feel versus something from this millenium.

http://www.immihelp.com/visas/sample-i94-form.pdf

http://www.immi.gov.au/managing-australias-borders/border-security/travel/passenger-cards/_pdf/english-ipc-sample.pdf
Yes it's an example of your point by dbIII · 2014-07-31 20:38 · Score: 1

Ultimately my example was about: "level of incompetence and lack of planning is strong in several levels", as you suggested but it was driven that way by the new vendor having far too much control over the situation and no risk to bear in the event of failure.
The government took them to court twice (outgoing and incoming - Queensland, Australia) and could not scratch that vendor (IBM) for any of the $500 million+ in estimated extra costs.
Put Obama's website guys on it by gelfling · 2014-07-31 22:55 · Score: 1

For another 840 million dollars they can probably get it to the point where only another 150 million is needed to get it running.
1. Re:Put Obama's website guys on it by Nethead · 2014-08-01 05:33 · Score: 1
  
  You owe me a new keyboard.
  
  --
  -- I have a private email server in my basement.