Slashdot Mirror
British Airways CEO Won't Resign, Says Outsourcing Not To Blame For IT Failure (bbc.com)
British Airways CEO Alex Cruz insisted he would not resign on Monday as he sought to draw a line under three days of chaos at the UK flag carrier after IT problems left tens of thousands of passenger stranded. In an interview -- the first since a global computer outage all but shut the airline down -- Cruz said he doesn't think "it would make much of use for me to resign." Separately, he also denied an outsourcing deal was to blame for the IT problems that hit on Saturday, causing the airline to cancel almost all its services over the weekend. From a report: A leaked staff email revealed Mr Cruz had told staff not to comment on the system failure. When asked about the email he told the BBC the tone was clear: "Stop moaning and come and help us." The airline is now close to full operational capacity after the problems resulted in mass flight cancellations at Heathrow and Gatwick over the bank holiday weekend. Questions remain about how a power problem could have had such impact, said the BBC's technology correspondent Rory Cellan-Jones. One theory was that returning systems were unusable as the data had become unsynchronised. [...] Cruz told the BBC a power surge, had "only lasted a few minutes," but the back-up system had not worked properly. He said the IT failure was not due to technical staff being outsourced from the UK to India.
We need communism now!
UNITE with the Campaign for a Free Internet because today, our future begins with tomorrow!
FOOD FIGHT!!
They pissed me off more than a decade ago, and I swore never to use their services again. Since then I flew all across the world, for scientific conferences, cooperation, or just fun. This includes even many flights to the US.
I'm not surprised BA sucks this bad, with a CEO like Alex Cruz.
"The agriculture ministry is not in charge of Gundam" - Japanese ministry official.
I remember when people would say that "X happened in Y with outcome Z", but here we don't get to know anything of what went wrong?
Not telling me in detail means I am highly unlikely to fly with them as they are seen as untrustworthy with something to hide.
Having a foreign IT staff isn't the issue, having an incompetent IT staff that is not able to manage the system and deal with issues like this is. If you are firing people who are able to do this and bringing in people who are barely able to hold stuff together because it lowers the salaries you pay then it is your own fault.
that's why their IT failed. In between sets of rowing machine powerlifting, and pleasing the "girls" with his powerful sexing, he'll fix that IT with a Python script and a slashdot post!
There should be a law that any company that uses cheap labour put the country in their company name and be required to pay unemployment benefits to workers they replaced. They should also be by law pay the Indian workers the same wage as british workers
to make sure that companies can't get a way with slave labour.
Any company that outsources should be required to have "We destroy jobs" on their website.
"He said the IT failure was not due to technical staff being outsourced from the UK to India."
So is this tacitly blaming the staff they still have?
You know what will fix your IT problems! More staffing cuts and outsourcing!
In the 1960s, they were developing home-grown technology in electronics (Ferranti) and computers, and contributed to a supersonic passenger transport plane.
Now they can't outsource technical work fast enough ... to be replaced by what? Rentiers and low skill manual work?
Be careful. You might get what you wish for. With all the scary anti-privacy laws, the UK is definitely heading in the "right" direction.
Nae king! Nae laird! Nae yurrupiean pressedent! We willna be fooled again!
How a single power supply could bring down the global operations of a major airline?? and this guy is the CEO???? WTF!
I don't think that it would make much of use for me, to resign.
There, fixed that for you. Commas are important.
Rome taught me patience and assiduous application to detail. Virtues which temper the boldness of great, general views.
The backup didn't work. Wow. Who woulda thunk? I suspect some IT person complained about the backup procedures being inadequate and he was probably fired. Someone else asked if they could actually test the backup and they were given a demotion.
And now that the backup failed to actually be a backup, we're all shocked and surprised (and it's definitiley not management's fault).
I tested my backups daily by importing the data into a different database. (Of couse, I'm an Oracle admin and am used to having failed backups).
Exactly. Who had the lobotomy?
Other feeble fudges are technical issues, contrived press releases.
This is a design issue. Did it exist before, or did someone change/omit/migrate something?
Because power failure is likely on any risk sheet, the consequences catastrophic, somebody approved a big red very high risk box. They need to be fired.
If the outsourcers were handling the risk management matrix, their caste was too low to influence outcomes.
Dear British Airways Upper Management,
This is your fault. To avoid another incident, you will bring in the operations IT managers, who are quite frankly, much smarter than you. Then sit down and shut the fuck up and listen to the solutions that these managers already know about, and which will easily fix the problem.
It would be best if all fools, MBAs, accountants and other technical illiterates were excluded from that meeting. A lawyer or too, on the other hand, may be quite helpful.
Hint. The solutions cost money. Guess why they were never implemented. Bonus question! Guess how expensive an unplanned failure is going to be.
Cheers!
Please do not read this sig. Thank you.
Well, first, Something Bad happened. Then, everybody tried to figure out What Went Wrong, but nobody could, because anybody who could find their ass with both hands had been laid off and their jobs outsourced to a faraway land where everything has to be microscopically explained, perhaps starting with "Well, hydrogen is one proton and one electron" and build from there. Then, The Suits started screaming for blood, but nobody they were screaming at was even competent enough to come up with a cogent response beyond "We're looking into it". Then, the Uber-PHB said "It's not because we shipped all our jobs to the lowest bidder, and It's Not My Fault".
"Never attribute to malice that which is adequately explained by stupidity" -- or a combination of stupidity and greed.
Cruz thinks he can get people to pay BA prices while slashing costs back beyond budget airline levels. He had form on this with Iberia. Meals cut, added extras cut on long haul flights, crew on zero hour contracts who aren't being paid with cancelled flights and all the IT staff within Britain being fired. No staff give a shit, and why should they?
Fuck you Alex. I hope this kills BA off.
If IT cost cutting (and the lost of institutional knowledge, let alone staff moral) and outsourcing was not a hugh factor, then maybe it is incompetent management?
Outsourcing the CEO job to India will probably save even more money, and the new CEO will a also claim that the next disaster is alost unrelated to outsourceing.
Outsource the CEO position and fire Cruz; see how he likes it.
Bet you anything that some more IT people get fired, though. Just keep firing IT people until all of your IT problems go away, BA!
"Tell me doctor, with all of your defenses, are there any provisions for an attack by killer bees?"
Please do the needful and let me put my point.
Time to resign.
Oh, and insource. Your data is your most precious resource.
"Never attribute to malice that which is adequately explained by stupidity"
I still insist that everybody has that backwards
“He’s not deformed, he’s just drunk!”
"Adding something for the risk you run" is also known as taking out insurance. If you're not Ned Flanders you do it, and if you do it often enough you can usually afford to self-insure for anything less than total catastrophe and still come out ahead compared with the alternative of always paying for "something better". Would you recommend we all always pay MSRP or can you acknowledge that sometimes it is worth accepting the risk of buying from a third party to get a discount?
Flanders wasn't wrong though - insurance is a gamble and like any kind you could lose if you place too many wrong bets. But the claim that you will always lose over time is BS.
He is right that "Outsourcing Not To Blame".
Power might of well have been the trigger, but the scale of the outage indicates much bigger problems afoot. The root cause of this turning out to be poor IT Governance, those big picture processes that prove the resilience is designed in, tested and proven. That is a failure of IT management.
IT treated as a cost centre, everything is outsourced on a lowest cost basis. Those suppliers are further whipped into line by crude metrics by managers that got a leg up by doing things quickly or cheaply, not properly. I see this kind of lack of concern for proper governance every day, address this lack of proper governance is by far the most difficult challenger I face as a consultant working in QA.
The NHS failure was exactly the same thing, the attack was the trigger, the root cause of the collapse of IT was governance failure by very senior management failure to ensure resilience was built in and proven.
IT is hard and how it works is invisible to those who don't understand it. BA might be screwed. Not only have they outsourced IT but it looks like they don't have the expertise anymore to even evaluate the quality of their IT or even prioritize and fund what their IT should be doing. So now not only is BA not good at IT they are doubly handicapped in that at least from their CEOs statements they can't even evaluate IT.
It makes me wonder if with the unusual frequency the airline industry is experiencing "computer glitches", that they are actually getting hacked and are trying to cover it up!
I didn't work at BA, but I did have a top view of the IT at two different airlines. I am not surprised of what happened. It seems that most airlines, in particular those that have been around for longer, rely on antiquated back end systems. While such systems may have been adequate 30 years ago, today we just expect these to handle more passengers with higher frequency interactions and higher reliability. These systems had to be scaled up, interlinked with all sorts of systems, and patched up numerous times by different contractors over the years. Since airlines often have to communicate with other airlines, e.g. to do code sharing or exchange frequent flyer point information, third parties service providers are often involved. This adds a lot of complexity, and in my opinion, little value at a steep cost. This global tangle of systems / languages / protocols ends up being very expensive to run and maintain. An airline isn't any more complex than another business with a similar number of employees. Probably within a year a team of ~5 software engineers could redesign everything from the ground up. Surely, a multi-year investment to gradual re-engineering the whole business logic would pay its dividends in reduced costs and increased reliability, right?
But which (IT) manager would approve such investment? Certainly not the ones I've met. If anything goes wrong in this system that they don't fully understand, they'd get the blame. On the other hand, if they subcontract to patch on new features onto something that was there before long ago, they can either blame the subcontractor or their predecessor. Better yet, the more complex the systems, the more secure their job and the more service providers that need to be involved. Besides, things have been working fine for the last 30 years, right?
He, along with almost all CEOs whine and bitch and moan about regulations, taxes, red tape, competition and so on when will they shut the fuck up, stop moaning about it and start helping?
What if his employers did to him what he says in his whinges? "If you don't change, we'll fuck off and take our output with us and you'll be stuck here wondering how you're supposed to wave your arms to get the plane in the air.
get Indian results
none whatsoever.
they obviously fucked up. he's going to blame it on somebody else.
we won't find out the truth until they fire the scapegoat who will then reveal all.
as soon as the non-disclosure agreement they signed to get their layoff package expires.
Absolute statements are never true
So, it looks like this Alex Cruz guy might survive as CEO. However in order to bolster his chances, he has to deny responsibility for the failure and cast it upon lower echelons. They will be cannon fodder for his continued career as CEO.
So what is a CEO, who is in ultimate charge of everything, to do? Pick one specific issue, claim the outage isn't connected to that specific issue, and carry on. Whether the outage was or was not connected to outsourcing is hardly the point, although for best political effect it ought not be connected. The facts can be helpful in this way.
However the bigger picture isn't about the facts, it's about appearance. The CEO needs to make sure that they can blame lower levels of the organization and suggest that "they were not informed, they are outraged both by the systems failure and the failure to be informed", and those involved have been disciplined or sacked. And of course this will Never Happen Again.
These comments aren't about BA specifically, nor their CEO. They might not apply to this case at all. However my experience tells me, the average business relevant systems failure works out this way.
Oh, there's incompetence here, but it's not the India that's the problem.
In my experience India has an incredible number of talented, capable people, but like talented capable people everywhere they cost more than ignoramuses. But even a country of a billion people has a finite pool of top-notch talent. On the other hand India does have an almost limitless supply of subpar talent, and Indian businessmen are enterprising to a fault. If a Western CEO jis willing to shell out good money for sub-par people, there's a killing to be made.
So who, exactly, is the fool in this scenario?
The British Airways debacle was an instance of a catastrophic failure being brought on by an unusually but statistically predictable event. Therefore, the new vendor the CEO brought in wasn't up to the job he hired them for. That's the CEO's fault, end of story.
The real problem is that people who are good at IT operations make their job look too easy. A fool looking at the lack of drama in a well-run data center is apt to mistake that for the job being easy.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
Next news, Alex Cruz announce a new name to reflect the new mindset of the company: British Groundways.
This is so true. The Instant Hero gets all the credit, where all the small preventative actions that keep the system oiled, consistent, and clean get mostly ignored. Make a list of preventative actions and mention them at your next performance review. You have to sell the idea that prevention is powerful.
And sometimes keeping a clean system requires saying "no" when somebody wants something custom or special without any real justification beyond "we like it that way" or "don't question my authority". Those who dole out special favors and ego-enhancing customization are often rewarded even if they complicate the system in the longer run.
Table-ized A.I.
In the 1980s, I interviewed for an entry level graduate position with BA, to work with their IT systems. Their people were arrogant and ignorant. When I asked them what languages their systems were written in, the HR droids had no clue. I never got a call back and I am very glad about that. Although the corporate culture has probably changed many times since then, it seems that their attitudes have not improved any. The fact that the head honcho will not take responsibility now is no surprise. I bet he will keep on taking his over-inflated salary and bonuses though.
They should have outsourced all their management including their CEO to India. Why not just let Indian Airlines run, manage, fly, and maintain the British Airlines. Might as well just call the whole thing Indian Air while we are at it.
Outsourcing is the belief that you can get someone else who supposedly does a job better than you to continue to work for you and not eventually switch sides and put you out of business.
At least we get a comfy chair!
It seemed weak but produced a strong G3-class geomagnetic storm, several other outages occurred as well...but you don't see even a hint of it in any MSM.
it had better be related to me not receiving my fucking bag!
FIGURE IT OUT, Fuck head! It's a bag! I need it! I'm in fucking Ghana for fuck's sake!
The company I worked for is outsourcing... They replaced 10 people with 40, the 10 people were highly educated and experienced local staff. The 40 people are two grades below the original staff and inexperienced with moderate education. This is how the outsourcing company intends to make money, by reducing employment costs. Now the new staff are utterly clueless on several subjects, notably what actually happens on the data center floor, how things are plugged together and how they should be plugged together properly. After a 3 month ongoing delay hooking up a few servers.. they finally called in an experienced local staffer to finally resolve it. 6 hours later, all done. The issues identified showed that they had been unplugging active connections and patching in the new ones, overwriting network configs without backing them up, misconfiguring network switch interfaces... even getting stuck on basic things like switch interface administratively down and wondering why its not working... basically the inexperience and poor decision making was slowly reducing redundancy on the network and distributed nature of the new teams meant that knowledge of those changes was too distributed for anyone to put it together what was happening.. eventually this would lead to an outage. See OP. Outsourcing companies are in a race to the bottom. The sales guys need to undercut, so they under spec the employees... its cut throat and it's irresponsible. The fact is, the CEO in this case was warned of the risks, giving something to someone else to manage for less money introduces a swathe of new risks that no clause in a contract will mitigate, only penalise. Your company is still going to experience reputational damage. Preaching to the choir.
...we have to cancel your flight, but we are experiencing computer problems and our computer staff is currently asleep. We will contact them as soon as it is working hours in India.
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
It would be delightful Karma if this is a result of the "training staff" conveniently forgetting to tell them of a devilish bug that they weren't allowed fix because there was a rather simple work around.
A work around that is documented somewhere. Now if they could just remember.
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
"Questions remain about how a power problem could have had such impact, said the BBC's technology correspondent Rory Cellan-Jones"
That might be a good question for Delta.
What are the requirements to become a BBC reporter? The media is completely inept these days.
Unfortunately, the British business system is designed around shielding the top management from any hint of responsibility.
It's the single biggest reason for the weakness of British industry as a whole. It's microscopically rare to see a CEO of a British company (regardless of the nationality of the CEO, it's the company that counts) actually accept responsibility for a failure.
In this respect they're quite unlike American or Japanese cultures. Europeans have some of the same disease, but not to the same extent. The country that most readily compares is Russia. It's all about protecting privilege.
Major problems caused by Indian IT. Should lose his job.
India has some talented people, but NONE of them work in IT.
Venezuela is an example of a single point of failure.
It doesn't matter what system of government they have, so long as they rely on oil the consequences of a price war with Saudis who can get oil out of the ground at less than one third of the price are kind of obvious.
Russia didn't cope well either so an oligarchy is not the answer either.
He said the IT failure was not due to technical staff being outsourced from the UK to India.
That may be true of the original cause of the outage, but what about the fact that it took 3 days to recover?
The real problem is incompetent people running things into the ground and getting bonuses and fat checks for it while getting away with lies and murder.
That's been my opinion/variant as well for anything above floor-clerk level.
An exhausted cashier on a double-shift at 3am fucks up.
A plan 6 months in the making signed off on by upwards of twenty people with the deliberate fuckery at its core can never be allowed such an excuse. It was exactly as intended.
Really?
Don't dumb yourself down - think about the process that failed (which was not anything to do with avionics, fuel allocation etc etc). What makes loading passengers into a aircraft far more difficult than emergency medicine?
Maybe they need to look into this cutting edge solution that the railroads and banking industry rely on - VAX/VMS distributed cluster computing. It might be just the ticket.
Perhaps if they didn't offshore the staff and/or outsource to a benefit-dodger agency, they might have some competent people.
There's plenty of Britons that would have done the job better, but the company makes the fatal mistake of offshoring.
Twitter supports and protects racists - by smearing their critics with the "Hate Speech" label.
Then, the Uber-PHB said "It's not because we shipped all our jobs to the lowest bidder, and It's Not My Fault".
What has Uber to do with this? And if they are involved in any way, we can be sure it's not just their fault, they did it intentionally.
What I see from the majority of commenters here is the lynch-mob mentality. Most people here are speaking without concrete evidence. It doesn't say anywhere in the article, or elsewhere, that this is due to Indians. Sure, they might be horrible coders and mightily incompetent. But that doesn't mean you lay the blame on them for every single fuck-up. If this is how the so-called educated cream thinks, I wonder what your average joe is capable of. However, it also speaks volumes about India's reputation when it comes to IT. I mean, there is no smoke without fire. A sad state of affairs on the whole.
Sooo... what probably happened is a UPS has been beeping unhappily begging someone to replace its battery somewhere for several years and nobody noticed until then...
I still want to know how a power cut of one locale of systems could take out their entire global network tho'... they'd better fix that quick because if their network is that vulnerable they are not going to have fun times if the script kiddies find their way in...
Excellent post thank you.
Money Junkies, personalize gains socialize losses is part of the first set of rules in their playbook.
Also to the earlier post " attribute to malice" junk is exactly that, junk. Again this logical conclusion fallacy statement is another attempt to pass the blame to someone else - same pattern.
It is very simple: When money is their only goal then all else suffers greatly.
Sounds like the CEO is trying to resist reaccomodation.
Maybe it took 3 days to re-hire the old guys. It could have been 1 day but they could not agree on the payment :-).
How did the moderators miss this post? It appears to contain actual details of the BA recovery failure!
"I've see a note with some details.
There were institutional issues with out of date software on network devices, and a L2 network stretched across both DCs.
The power outage caused a switch to fail, which had to be replaced and reconfigured from incomplete documentation. Multiple other devices failed to restart cleanly, due to the failed network, poor systems management and incorrect startup sequencing. The outsourcing staff took a "just reboot it" approach to failures.
From what I read, it took over 10 hours for the network to converge and stabilize, and even then application servers needed to be correctly sequenced.
The old in-house staff had not managed and documented existing systems which were not well designed, and the new outsourced staff did not have the skills and experience to recover the charlie-foxtrot."
Aeroflot might actually be a bit worse (prior to the weekend). Rossiya, however, has catered for me well.
For European airlines which are far better, look to Lufthansa, SAS, Finnair, and their ilk. For European airlines which are better, there's Air France, Olympic Air, and TAP. For European airlines which are slightly better, think of Easyjet and Ryanair (both cheap). Even Alitalia[*] is better.
In the last few trips, I flew Lufthansa, Finnair, SAS, JAL, and Air Canada. All are recommended for business class intercontinental travel.
[*] Always Late In Take-off, Always Late In Arrival.
I'm fairly lucky to be in a position where people get this. Our current team plans almost to a fault. Documentation gets done along with the work, and everything is well padded for time. It's boring around here. People are cheerful and nice. Sometimes people slide in late, and sometimes people slide out early.
Having worked in a "firefighter" shop before, where everyone was always scrambling to put out fires, I'm shocked by how much more we accomplish. On the few occasions where we have flare-ups, lots of people are available to assist, because we all have some capacity in our jobs. What's fascinating is that someone will say, "Well, I have about 4 hours of time I can give you over the next week, but I can't do more than that. Will that be enough?" Then you're trying to figure out if that's likely enough time to help, not enough to make a difference, and whether or not you need someone else.
Professional, no-drama, well planned out. There's a good chance that any new bosses will come from among our ranks, as that's been tradition for a decade or so. My only fear is if that doesn't happen. We're either going to continue being awesome, or it's downhill all the way. Not a lot of up to go from here.
Velociraptor = Distiraptor / Timeraptor
The real problem is that people who are good at IT operations make their job look too easy.
A good IT professional is like a god, if he is doing his job right you will never notice that he exists and everything keeps running like it always has. The only reason people have to interact with IT is that things stopped working and any instance of that happening should be extremely rare and short.
It's hardly headline news when a CEO doesn't accept responsibility for something
He said the IT failure was not due to technical staff being outsourced from the UK to India.
Was it....
Q: "Was the outage caused by the outsourcing of IT services to India?"
A: "No."
Or was it....
Q: "What was the cause of the outage?"
A: "The outage was not caused but the outsourcing of IT services to India."
Do you think the Wright brothers would have taken the insane decision to attempt powered flight if their computer was down? Planes can't fly without computers, and commercial air travel was impossible before the days of networked computers to handle the whole process from customer booking to landing the plane safely as a single invariable process.
And lying about it puts you in deeper shit.
Popular opinion says this less than weak explanation is not flying
Mod up.
Yes. Looks easy. Anyone can do it.
But when Mr CEO needs a surgeon - he goes shopping for the best.
The name of the game is to move on before being found out as a slash and burner.
Five months on, blaming oldies does not fly.
CEO's who give their IT department lobotomies should get to enjoy the same and wake up with an Irish - no make that an Indian accent.
People left are pissed, so he is wrong. ... they fire our friends, give us more work, and expect we will be happy? It became a toxic environment, so I left about 2 months later.
I've survived downsizing. A group I worked in cut 50% of our 20 person team. The amount of work was cut, but not enough. We were all told to work 15 hrs of unpaid overtime weekly (salaried people).
So
It isn't like skilled people can't find jobs.
CEOs can be really stupid, even if they are smart most of the time.