Slashdot Mirror
This Week 'IT Issues' Ground Delta Airlines' Flights (cnbc.com)
Delta Air Lines has been forced to cancel at least 150 flights, and expects to cancel even more. But "the IT department is working to rectify the situation as soon as possible," they tweeted Sunday -- more than four hours ago. Long-time Slashdot reader SonicSpike quotes CNBC:
Delta Air Lines U.S. domestic flights were grounded on Sunday evening due to automation issues, according to an advisory from the Federal Aviation Administration... "Delta teams are expeditiously working to fix a systems outage that has resulted in departure delays for flights on the ground," the airline said in the statement. "Flights in the air remain unaffected". [And their international flights were unaffected.]
Delta also grounded 2,000 flights last summer after a computer outage caused by a power outage in Atlanta. At the time Reuters reported that "Airlines will likely suffer more disruptions... because major carriers have not invested enough to overhaul reservations systems based on technology dating to the 1960s." And sure enough, just last week, another "IT issue" forced United Airlines to ground all their domestic flights.
Delta also grounded 2,000 flights last summer after a computer outage caused by a power outage in Atlanta. At the time Reuters reported that "Airlines will likely suffer more disruptions... because major carriers have not invested enough to overhaul reservations systems based on technology dating to the 1960s." And sure enough, just last week, another "IT issue" forced United Airlines to ground all their domestic flights.
How long can the airlines go on like this? Somewhere in office buildings around the country, there are MBAs and accountants working for various airlines who have compared the cost of in-house IT with the cost of outsourcing, and they all once decided that outsourcing was best. Somehow, I doubt they've included in their calculations the true frequency (and therefore cost) of IT failures that ground the entire airline for days. As these events stack up, these guys are going to have to re-evaluate their models for predicting the frequency and severity of failures, and at some point it's going to look like a good idea to have a real IT staff on-hand to keep systems working in the first place, and to deal with it when shit hits the fan.
This is simply a case of technical debt being piled on top technical debt. Don't blame anyone but management. Marketing screaming for more features and MBAs running a business that is a large technical enterprise. 50 years of added crap on top of crap and this is what you get, IT issues, outages, "power failures", automation issues. Each one causing tens of millions is losses. What the airline industry needs is a large IT colonic and then some good design to move forward.
You are not going to be able to fix this problem with the same thinking that got you into it.
"Somewhere, there's a computer that's "Preparing to configure Windows" after it rebooted in the middle of a flight scheduling run." ...And stuck in a reboot loop of "Windows Update Failed" and "Preparing Windows update."
We may never know what happened - but airlines are coming off their holiday change freezes so I'll bet one of their releases went bad. I don't work for Delta, but I do work in the industry, including some stints at airlines. The central problem is that airlines are incredibly low-margin businesses. Yes, they charge for everything and flying is expensive, but for every last minute $1000 ticket are hundreds of people demanding the same flight for $129 and getting it, even though they barely break even on those customers. Airlines' biggest costs are fuel, labor and planes. When it comes to IT, it's just massive amounts of technical debt built on a very old core set of systems. All that customer facing tech is really driving some ancient stuff several layers deep, collecting the information and presenting it in a nice format like your phone screen. All of these abstractions, wrappers on wrappers and middleware have to work perfectly and it's a rickety tower sometimes. Also, airlines are run by MBAs who don't consider their IT a "core competency", so it either gets minimal funding or dumped off on a contractor. Often, the contractors develop stuff like the phone app or one of a billion middleware components and there are always integration issues...but the people in charge love the ability to pay someone $x to implement "that phone thing" or "the ability to do X without talking to an agent."
With all the negatives, it's a very challenging and fun environment to work in for the right kind of person. I've been doing it for 20+ years and on balance I really like it. Resourceful types do very well in airline IT, as do IT geeks who understand and care about the business they're supporting. It's extremely frustrating at times as well, and there's way more firefighting than there should be. Typical businesses will just throw money at a problem until it goes away or they run out, which is why there are so many software tool vendors and expensive hardware systems out there. Go to an airline and tell them to spend 5 million bucks, and you can forget it unless it's required for compliance, safety related or guaranteed to return an immediate increase in revenue. Unfortunately this is where the technical debt comes from because there's never enough people, and all those people are running around putting out fires all the time. If I were working for Delta right now, I guarantee I wouldn't have been sleeping for the last 24 hours as everyone tried to figure out what had gone wrong.