Slashdot Mirror
British Airways CEO Won't Resign, Says Outsourcing Not To Blame For IT Failure (bbc.com)
British Airways CEO Alex Cruz insisted he would not resign on Monday as he sought to draw a line under three days of chaos at the UK flag carrier after IT problems left tens of thousands of passenger stranded. In an interview -- the first since a global computer outage all but shut the airline down -- Cruz said he doesn't think "it would make much of use for me to resign." Separately, he also denied an outsourcing deal was to blame for the IT problems that hit on Saturday, causing the airline to cancel almost all its services over the weekend. From a report: A leaked staff email revealed Mr Cruz had told staff not to comment on the system failure. When asked about the email he told the BBC the tone was clear: "Stop moaning and come and help us." The airline is now close to full operational capacity after the problems resulted in mass flight cancellations at Heathrow and Gatwick over the bank holiday weekend. Questions remain about how a power problem could have had such impact, said the BBC's technology correspondent Rory Cellan-Jones. One theory was that returning systems were unusable as the data had become unsynchronised. [...] Cruz told the BBC a power surge, had "only lasted a few minutes," but the back-up system had not worked properly. He said the IT failure was not due to technical staff being outsourced from the UK to India.
We need communism now!
UNITE with the Campaign for a Free Internet because today, our future begins with tomorrow!
They pissed me off more than a decade ago, and I swore never to use their services again. Since then I flew all across the world, for scientific conferences, cooperation, or just fun. This includes even many flights to the US.
I'm not surprised BA sucks this bad, with a CEO like Alex Cruz.
"The agriculture ministry is not in charge of Gundam" - Japanese ministry official.
I remember when people would say that "X happened in Y with outcome Z", but here we don't get to know anything of what went wrong?
Not telling me in detail means I am highly unlikely to fly with them as they are seen as untrustworthy with something to hide.
Having a foreign IT staff isn't the issue, having an incompetent IT staff that is not able to manage the system and deal with issues like this is. If you are firing people who are able to do this and bringing in people who are barely able to hold stuff together because it lowers the salaries you pay then it is your own fault.
that's why their IT failed. In between sets of rowing machine powerlifting, and pleasing the "girls" with his powerful sexing, he'll fix that IT with a Python script and a slashdot post!
"He said the IT failure was not due to technical staff being outsourced from the UK to India."
So is this tacitly blaming the staff they still have?
You know what will fix your IT problems! More staffing cuts and outsourcing!
Be careful. You might get what you wish for. With all the scary anti-privacy laws, the UK is definitely heading in the "right" direction.
Nae king! Nae laird! Nae yurrupiean pressedent! We willna be fooled again!
I don't think that it would make much of use for me, to resign.
There, fixed that for you. Commas are important.
Rome taught me patience and assiduous application to detail. Virtues which temper the boldness of great, general views.
The backup didn't work. Wow. Who woulda thunk? I suspect some IT person complained about the backup procedures being inadequate and he was probably fired. Someone else asked if they could actually test the backup and they were given a demotion.
And now that the backup failed to actually be a backup, we're all shocked and surprised (and it's definitiley not management's fault).
I tested my backups daily by importing the data into a different database. (Of couse, I'm an Oracle admin and am used to having failed backups).
Dear British Airways Upper Management,
This is your fault. To avoid another incident, you will bring in the operations IT managers, who are quite frankly, much smarter than you. Then sit down and shut the fuck up and listen to the solutions that these managers already know about, and which will easily fix the problem.
It would be best if all fools, MBAs, accountants and other technical illiterates were excluded from that meeting. A lawyer or too, on the other hand, may be quite helpful.
Hint. The solutions cost money. Guess why they were never implemented. Bonus question! Guess how expensive an unplanned failure is going to be.
Cheers!
Please do not read this sig. Thank you.
It would already be enough if their CEO has to personally pay for the blunder, and you'll see them replace cheap code chimps with sensible IT staff pretty fucking quickly.
CEOs don't give a shit about anything as long as it doesn't cut into their bottom line. And I mean their personal one, not the one of the company they are allegedly responsible for.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Well, first, Something Bad happened. Then, everybody tried to figure out What Went Wrong, but nobody could, because anybody who could find their ass with both hands had been laid off and their jobs outsourced to a faraway land where everything has to be microscopically explained, perhaps starting with "Well, hydrogen is one proton and one electron" and build from there. Then, The Suits started screaming for blood, but nobody they were screaming at was even competent enough to come up with a cogent response beyond "We're looking into it". Then, the Uber-PHB said "It's not because we shipped all our jobs to the lowest bidder, and It's Not My Fault".
"Never attribute to malice that which is adequately explained by stupidity" -- or a combination of stupidity and greed.
What I can't explain is why he is STILL the CEO.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Cruz thinks he can get people to pay BA prices while slashing costs back beyond budget airline levels. He had form on this with Iberia. Meals cut, added extras cut on long haul flights, crew on zero hour contracts who aren't being paid with cancelled flights and all the IT staff within Britain being fired. No staff give a shit, and why should they?
Fuck you Alex. I hope this kills BA off.
Outsource the CEO position and fire Cruz; see how he likes it.
Bet you anything that some more IT people get fired, though. Just keep firing IT people until all of your IT problems go away, BA!
"Tell me doctor, with all of your defenses, are there any provisions for an attack by killer bees?"
Maybe it wasn't a single supply, but rather a series of failures over the last 6 months which never got fixed.... and then the last one failed as well.
Please do the needful and let me put my point.
Time to resign.
Oh, and insource. Your data is your most precious resource.
"Never attribute to malice that which is adequately explained by stupidity"
I still insist that everybody has that backwards
“He’s not deformed, he’s just drunk!”
Maybe it wasn't a single supply, but rather a series of failures over the last 6 months which never got fixed.... and then the last one failed as well.
Or some forgot to plug in the monitoring cables for the redundant power supplies? I worked at a company where a hallway suddenly smelled like an open sewer for several weeks. What made it mysterious was that no sewer line went through that part of the building, leaving the building owner and plumber puzzled about the source. The smell came from leaking batteries inside a redundant UPS in the network closet on the other side of the wall. Since the monitoring cable wasn't plugged in, the one-man IT department didn't know that the UPS stopped working a long time ago.
He is right that "Outsourcing Not To Blame".
Power might of well have been the trigger, but the scale of the outage indicates much bigger problems afoot. The root cause of this turning out to be poor IT Governance, those big picture processes that prove the resilience is designed in, tested and proven. That is a failure of IT management.
IT treated as a cost centre, everything is outsourced on a lowest cost basis. Those suppliers are further whipped into line by crude metrics by managers that got a leg up by doing things quickly or cheaply, not properly. I see this kind of lack of concern for proper governance every day, address this lack of proper governance is by far the most difficult challenger I face as a consultant working in QA.
The NHS failure was exactly the same thing, the attack was the trigger, the root cause of the collapse of IT was governance failure by very senior management failure to ensure resilience was built in and proven.
IT is hard and how it works is invisible to those who don't understand it. BA might be screwed. Not only have they outsourced IT but it looks like they don't have the expertise anymore to even evaluate the quality of their IT or even prioritize and fund what their IT should be doing. So now not only is BA not good at IT they are doubly handicapped in that at least from their CEOs statements they can't even evaluate IT.
It makes me wonder if with the unusual frequency the airline industry is experiencing "computer glitches", that they are actually getting hacked and are trying to cover it up!
none whatsoever.
they obviously fucked up. he's going to blame it on somebody else.
we won't find out the truth until they fire the scapegoat who will then reveal all.
as soon as the non-disclosure agreement they signed to get their layoff package expires.
Absolute statements are never true
Oh, there's incompetence here, but it's not the India that's the problem.
In my experience India has an incredible number of talented, capable people, but like talented capable people everywhere they cost more than ignoramuses. But even a country of a billion people has a finite pool of top-notch talent. On the other hand India does have an almost limitless supply of subpar talent, and Indian businessmen are enterprising to a fault. If a Western CEO jis willing to shell out good money for sub-par people, there's a killing to be made.
So who, exactly, is the fool in this scenario?
The British Airways debacle was an instance of a catastrophic failure being brought on by an unusually but statistically predictable event. Therefore, the new vendor the CEO brought in wasn't up to the job he hired them for. That's the CEO's fault, end of story.
The real problem is that people who are good at IT operations make their job look too easy. A fool looking at the lack of drama in a well-run data center is apt to mistake that for the job being easy.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
This is so true. The Instant Hero gets all the credit, where all the small preventative actions that keep the system oiled, consistent, and clean get mostly ignored. Make a list of preventative actions and mention them at your next performance review. You have to sell the idea that prevention is powerful.
And sometimes keeping a clean system requires saying "no" when somebody wants something custom or special without any real justification beyond "we like it that way" or "don't question my authority". Those who dole out special favors and ego-enhancing customization are often rewarded even if they complicate the system in the longer run.
Table-ized A.I.
In the 1980s, I interviewed for an entry level graduate position with BA, to work with their IT systems. Their people were arrogant and ignorant. When I asked them what languages their systems were written in, the HR droids had no clue. I never got a call back and I am very glad about that. Although the corporate culture has probably changed many times since then, it seems that their attitudes have not improved any. The fact that the head honcho will not take responsibility now is no surprise. I bet he will keep on taking his over-inflated salary and bonuses though.
I've see a note with some details.
There were institutional issues with out of date software on network devices, and a L2 network stretched across both DCs.
The power outage caused a switch to fail, which had to be replaced and reconfigured from incomplete documentation. Multiple other devices failed to restart cleanly, due to the failed network, poor systems management and incorrect startup sequencing. The outsourcing staff took a "just reboot it" approach to failures.
From what I read, it took over 10 hours for the network to converge and stabilize, and even then application servers needed to be correctly sequenced.
The old in-house staff had not managed and documented existing systems which were not well designed, and the new outsourced staff did not have the skills and experience to recover the charlie-foxtrot.
At least we get a comfy chair!
It seemed weak but produced a strong G3-class geomagnetic storm, several other outages occurred as well...but you don't see even a hint of it in any MSM.
it had better be related to me not receiving my fucking bag!
FIGURE IT OUT, Fuck head! It's a bag! I need it! I'm in fucking Ghana for fuck's sake!
The company I worked for is outsourcing... They replaced 10 people with 40, the 10 people were highly educated and experienced local staff. The 40 people are two grades below the original staff and inexperienced with moderate education. This is how the outsourcing company intends to make money, by reducing employment costs. Now the new staff are utterly clueless on several subjects, notably what actually happens on the data center floor, how things are plugged together and how they should be plugged together properly. After a 3 month ongoing delay hooking up a few servers.. they finally called in an experienced local staffer to finally resolve it. 6 hours later, all done. The issues identified showed that they had been unplugging active connections and patching in the new ones, overwriting network configs without backing them up, misconfiguring network switch interfaces... even getting stuck on basic things like switch interface administratively down and wondering why its not working... basically the inexperience and poor decision making was slowly reducing redundancy on the network and distributed nature of the new teams meant that knowledge of those changes was too distributed for anyone to put it together what was happening.. eventually this would lead to an outage. See OP. Outsourcing companies are in a race to the bottom. The sales guys need to undercut, so they under spec the employees... its cut throat and it's irresponsible. The fact is, the CEO in this case was warned of the risks, giving something to someone else to manage for less money introduces a swathe of new risks that no clause in a contract will mitigate, only penalise. Your company is still going to experience reputational damage. Preaching to the choir.
He cut costs and made shareholder's money. Who cares about customer satisfaction...
Only the State obtains its revenue by coercion. - Murray Rothbard
...we have to cancel your flight, but we are experiencing computer problems and our computer staff is currently asleep. We will contact them as soon as it is working hours in India.
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
It would be delightful Karma if this is a result of the "training staff" conveniently forgetting to tell them of a devilish bug that they weren't allowed fix because there was a rather simple work around.
A work around that is documented somewhere. Now if they could just remember.
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
Venezuela is an example of a single point of failure.
It doesn't matter what system of government they have, so long as they rely on oil the consequences of a price war with Saudis who can get oil out of the ground at less than one third of the price are kind of obvious.
Russia didn't cope well either so an oligarchy is not the answer either.
Really?
Don't dumb yourself down - think about the process that failed (which was not anything to do with avionics, fuel allocation etc etc). What makes loading passengers into a aircraft far more difficult than emergency medicine?
Perhaps if they didn't offshore the staff and/or outsource to a benefit-dodger agency, they might have some competent people.
There's plenty of Britons that would have done the job better, but the company makes the fatal mistake of offshoring.
Twitter supports and protects racists - by smearing their critics with the "Hate Speech" label.
Then, the Uber-PHB said "It's not because we shipped all our jobs to the lowest bidder, and It's Not My Fault".
What has Uber to do with this? And if they are involved in any way, we can be sure it's not just their fault, they did it intentionally.
Shareholders.
They started to notice that dissatisfied customers inform others and create a HUGE backlash in social media, which in turn makes people turn their backs to the company. Which translates into lost sales.
And that's pretty tough for airlines where you have HUGE fixed costs and rather insignificant per-passenger costs. Every empty seat you're flying around hurts you. Why do you think they overbook like crazy?
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
I'm fairly lucky to be in a position where people get this. Our current team plans almost to a fault. Documentation gets done along with the work, and everything is well padded for time. It's boring around here. People are cheerful and nice. Sometimes people slide in late, and sometimes people slide out early.
Having worked in a "firefighter" shop before, where everyone was always scrambling to put out fires, I'm shocked by how much more we accomplish. On the few occasions where we have flare-ups, lots of people are available to assist, because we all have some capacity in our jobs. What's fascinating is that someone will say, "Well, I have about 4 hours of time I can give you over the next week, but I can't do more than that. Will that be enough?" Then you're trying to figure out if that's likely enough time to help, not enough to make a difference, and whether or not you need someone else.
Professional, no-drama, well planned out. There's a good chance that any new bosses will come from among our ranks, as that's been tradition for a decade or so. My only fear is if that doesn't happen. We're either going to continue being awesome, or it's downhill all the way. Not a lot of up to go from here.
Velociraptor = Distiraptor / Timeraptor
It's hardly headline news when a CEO doesn't accept responsibility for something