More Airline Outages Seen As Carriers Grapple With Aging Technology (reuters.com)

An anonymous reader writes: Airlines will likely suffer more disruptions like the one that grounded about 2,000 Delta flights this week because major carriers have not invested enough to overhaul reservations systems based on technology dating to the 1960s, airline industry and technology experts told Reuters. Airlines have spent heavily to introduce new features such as automated check-in kiosks, real-time luggage tracking and slick mobile apps. But they have avoided the steep cost of rebuilding their reservations systems from the ground up, former airline executives said. Scott Nason, former chief information officer at American Airlines Group Inc, said long-term investments in computer technology were a tough sell when he worked there. "Most airlines were on the verge of going out of business for many years, so investment of any kind had to have short pay-back periods," said Nason, who left American in 2009 and is now an independent consultant. The reservations systems of the biggest carriers mostly run on a specialized IBM operating system known as Transaction Processing Facility, or TPF. It was designed in the 1960s to process large numbers of transactions quickly and is still updated by IBM, which did a major rewrite of the operating system about a decade ago.

Comments

Dumb

[geek]

"Most airlines were on the verge of going out of business for many years, so investment of any kind had to have short pay-back periods,"

You really only see this type of thinking in the West. Most sensible companies know that when times are good, you build a war chest, when they are bad you invest the war chest to grow your business and be competitive. The problem wasn't that times were bad. You can always say times are bad. The problem was that they didn't make the best of things when times were good, and therefore deserve the cluster fuck situation they are in now.

Re:Dumb

[prograsm]

Agreed. Aging tech isn't the problem here, a complete inability to listen to or fund IT is the problem here. If they had a usable rolling backup system, it wouldn't matter how old everything is. If they had all brand new equipment and no functional load balancing system to compensate for outages that will always be a potential issue, they would still be offline for as long as it takes to fix everything. I have a hard time believing the words "off site redundancy" never came up in any IT budget meetings over the past half century, so their failures are 100% bad business decisions not IT issues. It would be no different if they had refused to budget for more fuel than exactly as much as predicted they would need. Tblaming the aircraft rather than the person that made the stupid decision to run out of fuel wouldn't make sense. It only works here because people don't understand IT, and the people that chose to allow outages like these aren't willing to admit it so they will repeat them again.

Re:Dumb

[clodney]

From what I have read, this was not an obvious WTF moment. Delta apparently has a complete disaster recovery facility with duplicate hardware. But they had a single point of failure in their infrastructure, which caused them to lose power to the entire datacenter, and everything went down. That part might be a WTF. But once they got everything booted up again, they had to contend with trying to get a system restarted that simply wasn't designed to ever fail completely. So it took hours to get all the pieces back up and communicating again.

Then their are the real world problems - flight A feeds into flight B, but flight A was late, meaning all those connections were missed and passengers have to be rebooked. And flight B can't fly anyway, because the plane is still sitting 500 miles away because the flight that would bring it to this airport was cancelled as well. And the flight crew that was supposed to bring flight B to this airport technically went on duty the moment they reached the airport, and now they have reached the max allowed hours in the day, so a new crew is needed. But that crew is in a different city...

This incident will span some fascinating failure analyses, and no doubt people will get fired and lawsuits will be filed. And like most DR scenarios, it is way harder in real life than it seems in planning and exercises. I wouldn't be surprised if this causes a big project to deal with outages and restarts, so that this doesn't happen as easily next time.

Re:Dumb

[Jeremiah Cornelius]

I worked on the first big web-enablement for AA's Sabre system, back in 97-99. Saabre was the key to inter-airline reservation scheduling. Travel agents used this as their main system, and some other folks around here may remember the gateway with CompuServ. eAAsySaber. LOL.

It was unlike any dotcom experience I had around that time. Super legacy. Impossible to change anything - and grave uncertainty that changes were even possible!

The Sabre core compute and data storage stack was built on a series of different mainframe and mid-range systems, back when instead of writing new business functions, you instead attached new business systems.

The glut of stuff crossed vendors occasionally. Mostly IBM. Parts dated to the 70's and through the 90's. I never met anybody who had a "mastermind" view of how it all worked. Instead, lots of analysts with diagrams - mostly from vendors and "big five" firms. Any proposed change had to be run through an exercise that called on the various experts in different parts of the system. Most were not so much expert, as "acquiring some expertise". ;-)

Our work became the basis for travel services like Expedia, and customer offerings by American Express Travel, etc.

I'm sure that this may have changed only somewhat. Saber was sold off, and became a core to Travelocity - who in turn were finally bought by Expedia, who consumed Saber information. Behind it all, there are a 360 and some front-end processors ported to AS400 systems, I'm nearly certain.

Re:Dumb

[lgw]

No one credible would count duplicate equipment in the same data center to be any kind of DR plan at all. That's like confusing RAID with backup. And just like you don't have a backup unless you've tested it, you don't have a DR plan unless you've tested it.

But a "disaster plan" needn't be limited to IT in any way. Air France had some sort of computing disaster recently, a similarly total outage, but they completed all their scheduled flights (not on schedule, but still). They had a disaster plan involving everyone behind a counter at an airport on the phone to a massive call center, where everything was verified "manually" from offline backup systems (and possibly print-outs). "Is Joe Slashdotter booked for flight 123?" "Give me a minute - yup, let him on the plane." Low tech, but it worked.

Aging?

[sexconker]

What's wrong with aging tech? If most airlines are on TPF and TPF works and TPF is still maintained by IBM, what's the problem with TPF?
Something being old doesn't mean it's bad. Quite often, the reverse is true. The mainframe is still the king when it comes to reliability and transaction integrity, for example.

Re:If it isn't broke...

[LWATCDR]

The Delta outage was caused by a power outage. Seems like TPF is not the problem.
Considering how well this 1960s tech seems to be working replacing it may and doing it better may not all that easy.

I call BS

[wcrowe]

This is bullshit. Software does not "age" the same way that a car or a washing machine ages. The hardware can age, but the hardware can be replaced, and in this case we are talking about IBM software and hardware, which has a long-standing reputation for reliability and for maintaining backwards compatibility.

I think the more likely story is that the interfaces to these systems are being compromised. That's why it's happening, first at one airline, then another. Someone, somewhere is fucking around with the airlines' reservation systems.

I think these stories about "fires" and "aging" software is covering up for the fact that these systems are getting hacked. If people start to lose confidence in the systems they'll fly less or stop flying altogether.

Re:is it that complex?

[im_thatoneguy]

Reservation system could be implemented in chapter 10 of your first programming book. It seems trivial thing?

It's actually really really complex.

It's not just a "reservation system" where you lock out a ticketed space for X seconds until someone completes a transaction. You really have to view it as "The Company" when you're talking about airlines. Let's say a pilot has been in the air too long due to a delayed departure in New York. He hits his max flight time for the next 24 hours but he was scheduled to fly from his destination to another leg. So now you need to replace the pilot. Which pilot? Well there is a plane coming into the airport around the same time as the NY flight. But is that pilot rated to fly that same aircraft? Ok he is, great. But because 30 of the 300 passengers are going to miss their connections now because of the delayed arrival they need to be moved to different flights. But those flights are maxed out. So you have to bump some passengers on a scheduled flight and move them to a later flight as well. Because the plane is getting in late it's also going to depart late. So you also need to either arrange all of the passengers at the next destination to be on different flights and set of a chain reaction or you need to pull in a different plane at the 2nd destination to short circuit the chain reaction. But where can you get a plane from for the cheapest? And how much will it cost to put people up in a hotel vs flying an extra crew in on overtime?

This is all simple enough to calculate with like 1-2 planes. But when you have 1,000 aircraft and all of the seat assignments effectively being interdependent along with business interests (profit/loss of changes), customer service interests such as ticket class... and you have to stay up to date instantaneously with dozens of terminals all trying to do the same thing manually in addition to the automatic callbacks for unexpected events... it's big engineering effort to not create some sort of automatic-trading style feedback loop that accidentally sets off a chain reaction that cancels every flight in the country.

Every change has a cost. No human can orchestrate thousands of interdependent variables with millions of passengers manually. You have to have a central director system which instantaneously handles all of the callbacks and dependencies for a change throughout the entire graph.

It's actually very cool when you stop and think about how well it does at keeping everything relatively straight.

Ugh

[sootman]

Do reporters even read these stories as they are writing them?!? "Airlines will likely suffer more disruptions like the one that grounded about 2,000 Delta flights this week because major carriers have not invested enough to overhaul reservations systems based on technology dating to the 1960s... [TPF] is still updated by IBM, which did a major rewrite of the operating system about a decade ago."

Big, complicated system, written by a big, experienced company, still maintained... Do they think we'd be better off if it were rewritten from the ground up as a Ruby on Rails app or something?

Psst, I don't want to cause a panic, but I heard that large, important chunks of the Internet run UNIX, which also dates back to the '60s.