More Airline Outages Seen As Carriers Grapple With Aging Technology (reuters.com)

An anonymous reader writes: Airlines will likely suffer more disruptions like the one that grounded about 2,000 Delta flights this week because major carriers have not invested enough to overhaul reservations systems based on technology dating to the 1960s, airline industry and technology experts told Reuters. Airlines have spent heavily to introduce new features such as automated check-in kiosks, real-time luggage tracking and slick mobile apps. But they have avoided the steep cost of rebuilding their reservations systems from the ground up, former airline executives said. Scott Nason, former chief information officer at American Airlines Group Inc, said long-term investments in computer technology were a tough sell when he worked there. "Most airlines were on the verge of going out of business for many years, so investment of any kind had to have short pay-back periods," said Nason, who left American in 2009 and is now an independent consultant. The reservations systems of the biggest carriers mostly run on a specialized IBM operating system known as Transaction Processing Facility, or TPF. It was designed in the 1960s to process large numbers of transactions quickly and is still updated by IBM, which did a major rewrite of the operating system about a decade ago.

Comments

Dumb

[geek]

"Most airlines were on the verge of going out of business for many years, so investment of any kind had to have short pay-back periods,"

You really only see this type of thinking in the West. Most sensible companies know that when times are good, you build a war chest, when they are bad you invest the war chest to grow your business and be competitive. The problem wasn't that times were bad. You can always say times are bad. The problem was that they didn't make the best of things when times were good, and therefore deserve the cluster fuck situation they are in now.

Re:Dumb

[prograsm]

Agreed. Aging tech isn't the problem here, a complete inability to listen to or fund IT is the problem here. If they had a usable rolling backup system, it wouldn't matter how old everything is. If they had all brand new equipment and no functional load balancing system to compensate for outages that will always be a potential issue, they would still be offline for as long as it takes to fix everything. I have a hard time believing the words "off site redundancy" never came up in any IT budget meetings over the past half century, so their failures are 100% bad business decisions not IT issues. It would be no different if they had refused to budget for more fuel than exactly as much as predicted they would need. Tblaming the aircraft rather than the person that made the stupid decision to run out of fuel wouldn't make sense. It only works here because people don't understand IT, and the people that chose to allow outages like these aren't willing to admit it so they will repeat them again.

Re:Dumb

[clodney]

From what I have read, this was not an obvious WTF moment. Delta apparently has a complete disaster recovery facility with duplicate hardware. But they had a single point of failure in their infrastructure, which caused them to lose power to the entire datacenter, and everything went down. That part might be a WTF. But once they got everything booted up again, they had to contend with trying to get a system restarted that simply wasn't designed to ever fail completely. So it took hours to get all the pieces back up and communicating again.

Then their are the real world problems - flight A feeds into flight B, but flight A was late, meaning all those connections were missed and passengers have to be rebooked. And flight B can't fly anyway, because the plane is still sitting 500 miles away because the flight that would bring it to this airport was cancelled as well. And the flight crew that was supposed to bring flight B to this airport technically went on duty the moment they reached the airport, and now they have reached the max allowed hours in the day, so a new crew is needed. But that crew is in a different city...

This incident will span some fascinating failure analyses, and no doubt people will get fired and lawsuits will be filed. And like most DR scenarios, it is way harder in real life than it seems in planning and exercises. I wouldn't be surprised if this causes a big project to deal with outages and restarts, so that this doesn't happen as easily next time.

Re:Dumb

[Zontar_Thing_From_Ve]

You really only see this type of thinking in the West.

While there is certainly some component of that, it's not the major reason why things are as they are. Airlines in the USA are not owned or managed by the government. If it really came down to this, the US government might let all them go out of business and let new airlines be built out of the ashes. Switzerland did that. Plus, some airlines are actually owned by the government in the countries where they are based or the relationship is not all that independent. Air France, for example, is theoretically an independent company, but if they were going out of business the French government would surely step in and save them despite it violating EU laws to do so. The US airlines know that the government may not have their backs like they did the automobile industry. Plus, being publicly traded on the US stock market is probably for them a bad thing. This causes them to make decisions for short term profit to keep the stock price high. Finally, in the USA most customers who fly in coach only care about price and literally everything else is negotiable. They will make decisions on price alone. This puts pressure on the majors US carriers to compete at perhaps unrealistic price levels with the smaller airlines, which no doubt reduces money available to spend to upgrade old computer systems.

Re:Dumb

[DerekLyons]

"Most airlines were on the verge of going out of business for many years, so investment of any kind had to have short pay-back periods,"

You really only see this type of thinking in the West. Most sensible companies know that when times are good, you build a war chest, when they are bad you invest the war chest to grow your business and be competitive.

You really only see this kind of simpleminded thinking in people who don't know what they're talking about - but who can repeat something they read somewhere like a parrot.
 

The problem wasn't that times were bad. You can always say times are bad. The problem was that they didn't make the best of things when times were good, and therefore deserve the cluster fuck situation they are in now.

The problem is, times have never really been good for the airlines for any extended period. An airline is a capital intensive business, and runs on paper thin margins. Historically, as soon as they get their head above water, they get pushed under again. The shift to jets in the 50's, the fuel shocks and deregulation in the 70's, recession in the early 80's, another recession in the early 90's, a race to the bottom in fares sparked by the rise of internet ticketing, the post 9/11 drop in business, the Great Recession of 2007... (just to hit the high spots) There's a reason why practically every major airline has ended up bankruptcy court at least once.

Re:Dumb

[RabidReindeer]

That's because in the West, the post-1980 business philosophy has been "Efficiency to the Max!"

You are expected to give "110% percent". Everyone is running Big Data analytics. The bean-counters scour the numbers and do cherry-picking to get the big profits and lemon-dropping to discard the losers. JIT inventory. On-demand elastic clouds. And all in the context of the next quarter's earnings report.

No sane general would commit troops without maintaining reserves for the inevitable unforeseen, but modern western businesses cannot stand a moment of wasteful "idle time" or resources sitting around unused.

And so, when the inevitable happens - train wreck. There's no spare parts, no idle people to put to work, nothing. No reserve capacity.

As they used to say back when computers were expensive objects of reverence, "Never before in human history has it been possible to screw up so badly on so large a scale so quickly". Such is the 2-edged sword of modern technology.

Re:Dumb

[dgatwood]

From what I have read, this was not an obvious WTF moment. Delta apparently has a complete disaster recovery facility with duplicate hardware. But they had a single point of failure in their infrastructure, which caused them to lose power to the entire datacenter, and everything went down. That part might be a WTF.

No, the WTF is not that the datacenter had a single point of failure. If their IT setup had been designed properly, that would have been a minor inconvenience. The WTF is that they didn't have at least three datacenters in geographically isolated locations with hot failover and regularly test the hot failover to ensure that it worked reliably and quickly in the event of a sudden, catastrophic loss of their primary datacenter.

Aging?

[sexconker]

What's wrong with aging tech? If most airlines are on TPF and TPF works and TPF is still maintained by IBM, what's the problem with TPF?
Something being old doesn't mean it's bad. Quite often, the reverse is true. The mainframe is still the king when it comes to reliability and transaction integrity, for example.

Re:If it isn't broke...

[LWATCDR]

The Delta outage was caused by a power outage. Seems like TPF is not the problem.
Considering how well this 1960s tech seems to be working replacing it may and doing it better may not all that easy.

That's greed for you

[H3lldr0p]

Always short sighted and thinking tomorrow will be the same as today.

What I'm afraid of is this business / investment / management continues to infect the rest of the world. I can't wait until all of the stock markets are controlled by algorithmic trading, with the next quarter's number the sole goal.

I call BS

[wcrowe]

This is bullshit. Software does not "age" the same way that a car or a washing machine ages. The hardware can age, but the hardware can be replaced, and in this case we are talking about IBM software and hardware, which has a long-standing reputation for reliability and for maintaining backwards compatibility.

I think the more likely story is that the interfaces to these systems are being compromised. That's why it's happening, first at one airline, then another. Someone, somewhere is fucking around with the airlines' reservation systems.

I think these stories about "fires" and "aging" software is covering up for the fact that these systems are getting hacked. If people start to lose confidence in the systems they'll fly less or stop flying altogether.