Delta Computer Glitches Force Flight Halts Third Year In a Row (bloomberg.com)

An anonymous reader quotes a report from Bloomberg: The U.S. airline grounded all domestic flights Tuesday to deal with a technology issue that affected some of its systems. About an hour later, Delta said it had restored all its systems, allowing the services to resume. While the carrier said there were no disruptions or safety issues with any flight, the systems failure was the third in as many years that forced Delta to shut its operations. In January last year, a 2 1/2-hour computer breakdown grounded domestic flights. Delta's worldwide computer systems failed in August 2016, causing massive cancellations. This time, international flights weren't affected, and the grounding was relatively short. Still, with limited updates on flight schedules, irate customers took to social media.

Comments

And I'm sure their CEO will get a massive raise

[Indy1]

while cutting infrastructure spending and outsourcing IT positions in a mad attempt to make his/her stock options go through the roof.

Rinse, Lather, Repeat

Still not surprised, but a little confused

[imidan]

I'm not surprised, although I keep thinking that the massive amounts of money Delta loses due to IT failures would be enough incentive to bring their IT infrastructure back under their control. Since 2016, Delta's CIO, Rahul Samant, has been working to move IT employees out of Delta into Indian IT consultancies (see http://www.fox9.com/news/delta...).

It seems to me that airlines basically have four things of real value: aircraft, ground crews to maintain the aircraft, flight crews to operate the aircraft, and an IT system that allows them to schedule and dispatch the aircraft. When an airline talks about its "core competencies," I feel like their IT system should definitely be one, because if it fails, the planes don't fly. That seems like enough reason to not offshore that part of the business.

Re: Still not surprised, but a little confused

[Cochonou]

Does a one hour general grounding cost so much ? In terms of image for sure, but in terms of money I am not so sure... Who is actually going to get a refund for this ? One to two hours delays are not rare.

Re: Still not surprised, but a little confused

[mhotchin]

Knock on effects. Oops, missed your connecting flight. Oops, flight crew is now over hours, have to get another one. Oops, flight didn't make it, so the flight out of the destination is hooped as well.

Re: Still not surprised, but a little confused

[Anonymous Coward]

"Delta's CIO, Rahul Samant, has been working to move IT employees out of Delta into Indian IT consultancies"

Oh surprise, surprise, surprise - yet another Indian IT boss who refuses to hire anyone who is not Indian. I guess blatant employment discrimination is a-okay just so long as it's the majority population that's getting shafted.

Re: Still not surprised, but a little confused

[dcw3]

Yeah, yeah. But in the big scheme of things, how much do you think it actually lost them? My guess is that it won't be but a fraction of a percentage in their quarterly report.

Re: Still not surprised, but a little confused

[DarkOx]

Don't forget if any busier airports are involved and you miss your window the ATC delay. Detla plays A LOT of games to make their ontime and cancellation numbers look a lot better than they are too. I have laterally had a 4pm flight be delayed for 8 hours while later flights to the same destination took off because it was better to screw the ~150 of us for the better part of a whole day to keep their later flights from figuring negatively into their ontime states.

Usually this does not happen because its the same plane being turned around and used for the later flights; but when its not watch out.

Re:Still not surprised, but a little confused

[bobbied]

I'm not surprised, although I keep thinking that the massive amounts of money Delta loses due to IT failures would be enough incentive to bring their IT infrastructure back under their control. Since 2016, Delta's CIO, Rahul Samant, has been working to move IT employees out of Delta into Indian IT consultancies (see http://www.fox9.com/news/delta...).

It seems to me that airlines basically have four things of real value: aircraft, ground crews to maintain the aircraft, flight crews to operate the aircraft, and an IT system that allows them to schedule and dispatch the aircraft. When an airline talks about its "core competencies," I feel like their IT system should definitely be one, because if it fails, the planes don't fly. That seems like enough reason to not offshore that part of the business.

How cute that you think airlines actually OWN the airplanes they fly or actually have the necessary people to fully maintain them. Many aircraft are leased and most of the maintenance work is done by contractors and many times on foreign soil. Also, the airline business is one of managing debt and cash flow, often it's a race to cut costs faster than your competition to keep fares low and profit margins up.

Re: Still not surprised, but a little confused

[whoever57]

Yeah, yeah. But in the big scheme of things, how much did they "save" by outsourcing IT?

Re: Still not surprised, but a little confused

[dcw3]

A shitton more than this will cost them, and there's no assurance that not outsourcing IT would have avoided this outage. Did you have a point, or just upset about outsourcing...that I get, but this outage is no big deal.

Re: Still not surprised, but a little confused

[commodore64_love]

In the 2017 software crash they had to cancel 150 flights. Figure $200 per set, that's 4.3 million dollars (gross loss). In 2016 they canceled over 1000 flights, so that's 28.8 million (gross).

- Since the planes didn't fly the actual loss is less (no fuel burned, no hours paid to stewards/pilots, etc).

Outsourcing to India reduces salary cost from $113,000 to $9000 per engineer. I don't know how many engineers Delta IT uses? 1000 maybe?

DELTA SAVES $104 million in labor costs.

Re:Still not surprised, but a little confused

[imidan]

How cute that you think airlines actually OWN the airplanes

Condescending tone aside, it's not that important what the financing mechanism is for the aircraft. Airlines have a lot of money tied up in inventory. Maybe most maintenance is done by foreign contractors, but it's also done following recommendations by the manufacturer and regulations by the FAA. Plus, it's real bad business when a passenger airplane falls out of the sky because the operator cheaped out on maintenance. Given the rarity of plane crashes due to poor maintenance (Wikipedia lists 8 or so in the US since the FAA has existed), it seems that the airlines are doing okay there.

the airline business is one of managing debt and cash flow

And my point is, while your planes are all grounded because you offshored a mission-critical part of your enterprise to a bunch of guys who are just about half-competent and have just about half a shit to give about your business, your cash flow is all going one way. In 2016, Delta IT systems had a 5 hour outage that cost $155 million. They've had fleet-grounding IT failures every year since then. As their infrastructure continues to decay under outsourcing, I don't see these outages getting shorter or less frequent.

Re: Still not surprised, but a little confused

[imidan]

Delta reported to investors that their 5-hour outage in 2016 cost $150 million. Whatever the numbers are, I guess they believe they save money pay not having in-house IT. It seems risky, though, if 5 hours costs you that much. What happens when your offshored contractors cause a 10 hour outage? I doubt that the cost of the outage scales linearly with time... I'd guess it goes up faster than that.

Re:Still not surprised, but a little confused

[bobbied]

And my point is you don't have much idea what the issues are when running an airline.

It's about strictly managing costs per passenger mile and cash flow because such businesses are heavily leveraged and have huge amounts of money flowing around, with very little of it actually profit. Where I'm sure they don't like the fact that their IT systems went down and I'm confident they are in the process of showing who ever ends up being found "responsible" the door, the sad truth here is that such cost cutting is absolutely necessary. And YES if they can eek out a few pennies per passenger by offshoring their IT department, you can bet they will do it.

As you correctly point out, they MUST maintain the aircraft pre FAA standards, and they simply cannot skimp on the mandated airworthiness of the aircraft they use, they also must comply with Crew training and qualification regulations or get fined in a big way. Airlines though MUST control costs any and every way they possibly can, which is why many contract out anything more than line maintenance and "A" checks. And if they are willing to have their aircraft inspected by some foreign country, you can bet they'd jump at offshoring IT services.

After all, the successful airlines have been in the business of cutting labor and services costs to the bone for decades. Why would you think IT isn't or shouldn't be on that list? Mission critical is kind of a funny thing, and if they've already saved more than the two hour delay cost them (which in the grand scheme of things wasn't all that much considering they suffer such delays for weather issues at their primary hubs all the time) those IT service outages don't look so bad to management.

"Yes, sir, that outage was bad and cost us $10 million, but we've saved $500 million in costs the last fiscal year alone doing this." (assuming one had the documentation to prove this) would be a trade ANY airline executive would take.

Not saying that happened, but it's clearly the trade off being made here. If they are making more money with the trade, that's a good thing.

Re:Still not surprised, but a little confused

[imidan]

I mean, as long as it's cheaper to suffer IT outages than it is to fund IT, clearly they'll keep offshoring. It doesn't look like a sustainable model to me. I understand they're going to do the more profitable thing, but when this tech debt catches up to them in the form of a catastrophic outage, will they be able to withstand the fallout? I'm sure they have risk management people who model this out and say the answer is yes, but those guys aren't infallible.

Re:Still not surprised, but a little confused

[bobbied]

It's how the MBA's make the world go round, rob Peter to pay Paul if you have to, but keep the stock price up... It's manage to quarter and bail out of a company just before Peter runs out of money.

Airlines have done that for as long as I can remember and I grew up with two parents who worked for a major carrier all their lives.

So, you may be right, but pushing IT services off shore is the least of their concerns and I'm sure they have performance and availability clauses in their IT supplier contracts. There are just some things you simply cannot offshore as an airline and they have run out of cost reduction options for many years. Now days, airline management is watching load factors, dispatch percentages of their equipment, and are gladly spending billions on aircraft to get 20% reductions in per passenger mile fuel costs. Profit margins are razor thin and ticket prices are hotly contested. So it may be worth it, at least in the short term to do all sorts of things that don't look good long term.

Delta is just another company on this line of fail

[Anonymous Coward]

Tell it to the credit agencies, tell it to banks, tell it to hotels, tell it to telcos, Apple/MS/Adobe/Cisco/etc. It's RARE to have a company do this stuff well, airlines are absolutely bled to death already. Delta is no Virgin or Lufthansa.

Face Recog poops the bed

[bmimatt]

Great! Let them bring in face recognition dependency into the mix and they're sure to prop up competition stock.

https://tech.slashdot.org/story/17/05/19/208223/delta-airlines-tests-facial-recognition-to-speed-up-baggage-check-in

Joe vs. the Volcano

[mentil]

They neglected the annual human sacrifice to the AS/400 gods.

Re:Joe vs. the Volcano

[dwywit]

Is that what they use? "Annual human sacrifice" AKA maintenance contract.

IME AS/400s sit in the corner working and getting dustier and dustier until management decide to stop paying annual maintenance.

"Geez, why are we paying this much every year? Damn things never go down, we can skip it this year"

And BINGO! HDD failure - which,as an ad-hoc service call, will cost ~{annual maintenance$$$} to fix.

Fly American

[thegarbz]

You have the choice of being punched in the face and having your guitar broken, or not getting off the ground in the first place.

https://www.youtube.com/watch?...

Re:Fly American

[NormalVisual]

Don't forget, having the police show up to drag you off the plane is an extra-cost service that's also available.

Re:Result of third world labor

[dwywit]

Jeez, I'm gonna have to start browsing at +2, if posts like this keep getting +1.

Hint: follow the money. It's *always* about the money.

Also,

[ReneR]

maybe do not base your business on Windows?

Re:Delta is just another company on this line of f

[omnichad]

And if you're already going to do a bad job, outsourcing starts to make sense. No realisation that it might have value to do a good job.

Yawn

[dcw3]

Really? A whole hour? So 1/8760th? Well, there goes their five nines.

Re:Yawn

[LordWabbit2]

Erm, none - granted it could have been written better, but maybe you just suck at English.

Re:Yawn

[LordWabbit2]

Sorry, replied to the wrong post, please ignore.

Re: WHINY NAZI WHINES ABOUT BEING UNEMPLOYABLE

[Highdude702]

YEA!! Let's all get indigenous!!

Re:Result of third world labor

[drinkypoo]

You USED to be a smart country with excellent education, but your culture is dragging you and your whole society down.

Thanks for proving that there's no shortage of ignorance available. Our education system was deliberately compromised by politicians who got elected due to gerrymandering, on behalf of ultra-wealthy owners of corporations who wanted to preserve their dominance over others. It has absolutely fuck-all to do with culture and everything to do with greed.

Re: Yearly IT room cleaning

[Anonymous Coward]

maybe some hotshot indian programmer was trying to shoehorn in the latest systemd upgrade

Re:English. What a shit language...

[LordWabbit2]

Erm, none - granted it could have been written better, but perhaps the problem is that you suck at English?