IT Crash Causes British Airways To Cancel All Flights (cnbc.com)

An anonymous reader quotes CNBC: British Airways canceled all flights from London's Heathrow and Gatwick airports on Saturday as a global IT failure upended the travel plans of tens of thousands of people on a busy U.K. holiday weekend. The airline said it was suffering a "major IT systems failure" around the world. Chief executive Alex Cruz said "we believe the root cause was a power-supply issue and we have no evidence of any cyberattack." He said the crash had affected "all of our check-in and operational systems." BA operates hundreds of flights from the two London airports on a typical day -- and both are major hubs for worldwide travel. Several hours after problems began cropping up Saturday morning, BA suspended flights up to 6 p.m. because the two airports had become severely congested. The airline later scrapped flights from Heathrow and Gatwick for the rest of the day.

Comments

a power supply failure??

[Anonymous Coward]

So a power supply failure can bring down all operations on a global scale. Good to know that BA had outsourced part of their IT staff to India!!!

outsourcing

[Anonymous Coward]

MBA to board: I've got a great idea to cut costs! It will save millions!

Somewhere, an IT guy is crying

[whoever57]

Somewhere, there is probably an IT guy who has been begging for the budget to upgrade some old machines, or move the services onto a cloud provider and was ignored.

He's crying today, because this huge revenue loss could probably have been avoided with a small budget for newer hardware or more redundancy.

Re:Somewhere, an IT guy is crying

[nadaou]

He's crying today, because this huge revenue loss could probably have been avoided with a small budget for newer hardware or more redundancy.

And despite that s/he knows who will take the blame for it.

Re:Somewhere, an IT guy is crying

[grasshoppa]

move the services onto a cloud provider

"Cloud" service providers have no place in mission critical roles by virtue that the "Cloud" is a faster way of saying "abdicating responsibility". If you make millions of dollars a day on the back of your IT infrastructure, then the last thing you do is outsource the responsibility of said infrastructure to a 3rd party company which has different priorities than you do.

Any IT manager making such a recommendation is a) lazy, b) useless and c) should be fired.

Re:Somewhere, an IT guy is crying

[AmiMoJo]

The only alternative is to spend vast amounts of money building your own redundant systems, which clearly BA were unwilling to do. Using cloud services makes perfect sense.

Take Amazon's cloud services as an example. To get that kind of reliability, with systems distributed around the world for responsive operation an redundancy you are going to need a large number of geographically distributed services and a team to look after them. A team that is available 24/7 with response times in minutes.

And you will still have the same local problems, like internet connection reliability, and the same development problems. You don't have to waste time and effort administering your own servers either, dealing with mundane stuff like HDD failures or managing 30 different datacentre operators.

Unless your company is willing to put a massive amount of effort into that stuff for some reason, it's dumb to even try.

Re:Somewhere, an IT guy is crying

[Kjella]

Unless your company is willing to put a massive amount of effort into that stuff for some reason, it's dumb to even try.

IAG (the holding group of British Airways) have a market cap of 13 billion GBP or about 17 billion USD, guesstimating by fleet size BA is almost half of that. I'd understand if you were talking about a fly speck of a company but an 8 billion dollar company can damn well run their own infrastructure without a cloud provider with geographical distribution, 24/7 available teams and all that.

Re: Somewhere, an IT guy is crying

[Anonymous Coward]

Netflix has a $70B market cap and they run in AWS and have no intention whatsoever to go back to traditional data centers. Running them well is hard and hiring competent people to run them is even harder: Amazon, Google, Facebook and Microsoft hired a good chunk of the talent. Zinga tried AWS, found it expensive so went back to traditional Datacenter and cambe back running to Amazon.
I have worked on several migrations to AWS and the dumb companies do it without changing their software architecture.

Newsflash, AWS is expensive if you do not rewrite your code for auto scaling and Setup your QA/Dev environments to be 'on demand'. But as far as uptime is concerned, you cannot beat Amazon uptime if you have built multi-region deployments. If you use it like a traditional data center, well shit happens and Amazon machines die like any other. Their SLAs actually guarantee nothing about individual hosts.

Re: Somewhere, an IT guy is crying

[fluffernutter]

Usually the problem is when they go in house they want their administrators to work for $15 an hour, and then when they can't find good ones and the systems fail, they throw their hands up and go back to paying way more for cloud services than proper admins would have cost in the first place.

Idiots in charge!

[Anonymous Coward]

The Mythical Man-Month was written in 1975. In a very detailed way, it described how common business-planning stategies fail when applied to information technology projects. But did anyone listen? We've known how to avoid these sorts of problems for over 40 years!

Re:Idiots in charge!

[gweihir]

"We" (as in people that actually have a clue what they are doing) have indeed known that. But the decision-makers have no such understanding. While it is really tacky, I have had to explain catastrophe scenarios to customers that would have killed their company, and all that was needed was a failed software functionality update (which they wanted to do without a possibility to roll-back and no working plan for keeping business going any other way). The people making the decisions these days are bean-counters with zero understanding of risk-management or "visionaries" that have even less of an understanding about the reality of things. And, unfortunately, this often is aided by a corporate culture of "don't rock the boat" and people that warn of consequences get silenced.

Expect more of these utterly pathetic failures.

Pilling up technical debt is utterly stupid

[gweihir]

Of course, it requires more than the myopic 3-month planning that most MBAs are capable of at maximum. It also requires a real understanding of risk management and staying away from all short-term optimization. Otherwise, you end up at "save a million, lose a billion", as this seems to be a fine example of.

Claiming this was a "power supply issue" is just lying by misdirection. The root cause is lack of redundancy, lack of resilience and lack of effective business continuity management. All things that cost money and that do not generate profit _unless_ something like this happens. In a healthy infrastructure, one (or even several) power supplies blowing up will not kill your ability to do business.

Events like that are almost universally due to gross mismanagement and should not only result in termination but also prosecution of the "leadership" that allowed this to happen by not being prepared.

Re:Other sources: IT outsourcing

[gweihir]

"Power supply failure" does not take down a well-designed and well-maintained infrastructure. This is just a smokescreen to hide incompetence.

Re:The major issue is...

[Anonymous Coward]

Funny, but the bigger issue is there anyone at Tata that was there the last time BA restarted their systems? At the bank I used to work at, we were replaced by contractors, and two years later when they restarted the zSystem, they found-out the hard way that no one knew what to do.

Re:Busy U.K. Holiday Weekend...

[Anonymous Coward]

The international language of business is English.

Did you know that England isn't in the US?

Either amature hour or a lie

[Murdoch5]

Massive world wide systems like this, should always have at least two entire working deployments, one kept in a down state and one kept up and working, that way if a problem happens, you just bring the second data center online and off you go.

If a power supply issue could bring down your entire system, you didn't design it correctly, PERIOD! If your entire system hinges on a single power supply failure, you ALWAYS have a second one on an alternative supply, in fact, you'd have multiple supplies to each data center, from different providers, just to make sure power issues can't cause these types of issue.

If the problem really comes down to a power supply, fire the IT department, fire the System Architects and start doing things properly.

Re:Either amature hour or a lie

[thegarbz]

Define "correctly". We can design and build things for any scenario. With unlimited money, and investors who don't care about a profitable business we can do ANYTHING. Blanket statements get you nowhere.

Give us up-time numbers, design goals, costs of failure, associated profits. Will BA report on their balance sheet a loss larger than the cost of hardening their entire infrastructure? Tune in on the 31st December this year to see how little designing things "correctly" matters in the corporate report.