Slashdot Mirror

← Back to Stories (view on slashdot.org)

Computer Glitch Friday Grounded US Airways Flights

Posted by timothy on from the when-will-southwest-come-to-knoxville dept.

mschaffer writes "A computer glitch Friday night snarled the travel plans of US Airways customers, as reports flooded in of flights grounded around the country." As someone stranded for several hours yesterday by this outage, "glitch" seems like quite a euphemism. With outgoing flights blocked, and new ones arriving full of passengers expecting to meet connections, the atmosphere got a little heated. Customers could see nice weather, and planes lined up outside, but "The System Is Down" trumps all. The E concourse at Charlotte (a US Airways hub) was packed full of customers ranging from livid (a handful) to merely angry (most) to calmly resigned — which means those of us with seats, snacks, and books or computers. It was disheartening to see how brittle is the infrastructure the airline employs; with the part of the system visible to airline employees down, customers thought they might get more information, or even rebooking, through the US Airways website. But that was down, too, and all the desk staff could do is shrug.

19 of 140 comments (clear)

  1. COBOL's fault, I'm sure! by Kensai7 · · Score: 4, Funny

    OK, let's count to three and blame COBOL! :p

    --
    "Sum Ergo Cogito"
  2. umm... by datapharmer · · Score: 2, Interesting

    that sucks. No backup paper system in place? Can't they just read what the tickets say such as flight and seat number? They know where the flights are going as most are routine. It seems they should have been able to get *some* flights in the air.

    --
    Get a web developer
    1. Re:umm... by linest · · Score: 3, Interesting

      You can't fly unless you can prove your aircraft has had all required maintenance done. There are also rules about the number of hours per day crew members are allowed to be in the air. I suspect these records could be printed and used if it were a planned outage but this wasn't.

    2. Re:umm... by DesScorp · · Score: 5, Informative

      I work in airport IT, so I'll describe what I see airline crews doing during trouble. If the system at one gate or terminal is down, yes, then they'll send the plane on it's way. This is called "boarding manually". They simply hand collect tickets, hand count bags, etc, and send the flight off. After they've gathered all of the info thats been collected manually, they'll send it to their local office or front desk and process it at working terminals that have a connection to airline systems. It's a pain, but do-able. But if EVERYBODY is down, then the whole thing grinds to a halt. If no one has any access to all the schedule info, weight and baggage, manifests, etc.... then it's simply impossible to board manually on a massive scale.

      --
      Life is hard, and the world is cruel
    3. Re:umm... by DerekLyons · · Score: 2, Informative

      No backup paper system in place?

      One of the reasons they went to computers in the first place is because paper systems could no longer handle the workload... And that was back in the 60's when air traffic volumes were a fraction of what they are today. I.E. having to maintain a duplicate paper system would actually slow things down likely without actually providing sufficient backup.
       

      Can't they just read what the tickets say such as flight and seat number?

      That isn't much help with getting the luggage on the appropriate aircraft. Nor does it help to inform what flights (that you're expecting passengers from) are on time or nearly so and will or will not effect the flight in question. (Let alone routing the luggage involved.) Not to mention the number of passengers and the weight of the luggage - something the pilot needs to know to operate the aircraft safely.
       

      They know where the flights are going as most are routine.

      There's a lot more information flowing through the system than just "plane A goes to destination B" and "butt X goes into seat Y". With the system down they don't even know when/where plane 'A' is in order to get butt 'X' onto it.
       

      It seems they should have been able to get *some* flights in the air.

      No offense, but that's because you don't even remotely understand the problem. (And seemingly can't even be bothered to try by asking questions rather than making statements.)

    4. Re:umm... by FlyingGuy · · Score: 3, Interesting

      In 1960 most everyone was running Sabre

      Most everything was connected to hard lines that went back to the big main frame machines that ran it.

      If one terminal was down then it was most likely the terminal that had failed or possibly one of hundreds of hard lines back to the Main Frame

      Now days with everything being all cloudy good luck figuring out what might still be available. It could have been something as simple as the single bit of fiber serving that main concourse was damaged or a router someplace in the airport had failed or some cloud vendors routing had gone south or hell it might have been something as dramatic as what happened to Amazon a little bit ago.

      --
      Hey KID! Yeah you, get the fuck off my lawn!
    5. Re:umm... by Chaos+Incarnate · · Score: 2

      They didn't have Homeland Security wiggling their fingers and watching the airlines and passengers dance like marionettes.

      --
      Benford's Corollary to Clarke's Law: "Any technology distinguishable from magic is insufficiently advanced."
    6. Re:umm... by Grishnakh · · Score: 2

      The number of passengers per day in 1940 was next to nothing, and today it's staggering.

      Similarly, back in 1940 or 1950, "long distance" telephone calls (remember those?) were routed manually, by operators at switchboards. Can you even imagine trying to go back to that, with the sheer number of "long distance" telephone calls going on these days?

      When you do things of great complexity in huge volume, you need to have computers do them. Sure, it's possible to do them manually if you already have an army of trained personnel at hand, and equipment and processes in place for them to use and follow. But it's not like you can just go back to that in an instant. Just like there are no more manual operator switchboards at telephone companies to fall back to, there's no way to fall back to handling flights manually.

      The problem is that many things that are automated simply aren't designed with sufficient redundancy and reliability. Some high-profile outages will fix that though.

    7. Re:umm... by digitig · · Score: 4, Insightful

      This simply isn't about the problem. It's about whether the solution has been implemented. Nothing more.

      And that is down to whether it is cost-effective to implement the solution. You will never be able to get the probability of failure down to zero. and the cost skyrockets the closer you get to zero. How often do outages like this happen, and how much would it cost to prevent them at every airport worldwide? And to prevent every other conceivable scenario? Yes, it could have been prevented, and lots of other possible outages that didn't happen could have been prevented, but the cost of air tickets would be prohibitive.

      --
      Quidnam Latine loqui modo coepi?
    8. Re:umm... by Deadstick · · Score: 3, Interesting

      Rent an amusing sf film made in 1953 called "The Magnetic Monster" and watch an airport official tell Richard Carlson "We can't search ALL the flights. This is Los Angeles International; we have over a dozen departures a day!"

      rj

  3. Re:Backups? by RdeCourtney · · Score: 2

    Because they've calculated the customer apathy and money lost is less than implementing backup procedures. Remember it's all about $.

    --
    Insert signature here...
  4. Wait, what? by Anonymous Coward · · Score: 2

    FTFA: "The Tempe, Ariz-based carrier cited a power outage near one of the airline's data centers in Phoenix as a possible cause."

    A POWER OUTAGE?! So, no UPSes, no generators, and no multiple utilities at a main data center for a major company? Come on now...

  5. Could you tell the difference? by Cutriss · · Score: 4, Interesting

    As I am now located in proximity to an airport with a US Airways "service focus" and have had the "pleasure" of flying with them several times, I have to ask - how would you be able to tell the difference? Every time I've been in a US Airways terminal, there's always a significant number of non-weather-related delays and cancellations (compared to the other airlines' monitors). My wife and I have independently had three separate incidents this year where we were 4th and inches from having to stay overnight at an airport due to cancellations/late planes/overbooked crew/etc. In two of those cases, I had flights where we took off at the 2'55" mark, just shy of the three hour requirement to return to gate and let everyone off. The cynic in me suspects that US Airways is actually using that three hour window to plan its flights.

    It's an abhorrent mess, and when I see the US Airways CEO defending against his last place customer service ranking, I have to wonder just how much denial one management team can stand.

    --
    "Mod, mod, mod...and another troll bites the dust."
  6. Re:Very interesting by brusk · · Score: 3, Insightful

    Would you buy tickets that cost $25 than anywhere else more because the airline advertised redundancy in its IT systems? People choose flights based on price (secondarily, on frequent flier plans, etc.), but how is a consumer supposed to choose based on factors like this, except in the most general terms (on-time percentage)? The airline management knows this, and would be silly to invest too much in things that will raise costs without enabling them to increase revenues.

    --
    .sig withheld by request
  7. Whatever did we do... by Anonymous Coward · · Score: 2, Insightful

    ...before computers came along and made our lives so much -easier-?

    1. Re:Whatever did we do... by 246o1 · · Score: 3, Insightful

      ...before computers came along and made our lives so much -easier-?

      Not fly anywhere, because it was too expensive.

      --
      Although the moon is smaller than the earth, it is farther away.
    2. Re:Whatever did we do... by DerekLyons · · Score: 2

      Back in the 60's when airlines started computerizing, air traffic volume was a fraction of what it is today. You'd have to be nearly fifty to have even been alive at a time when computers weren't starting to make our lives easier in a variety of ways. ("Computers" !=" PC's".)
       
      I remember trying to make airline reservations back before the web. You couldn't pay me enough to go back to those days. (Unless you could also give me my twenty year old body as well.)

  8. Re:Backups? by linest · · Score: 2

    Because they've calculated the customer apathy and money lost is less than implementing backup procedures. Remember it's all about $.

    I'm not crazy about the way that's phrased, but you are essentially correct. Establishing backup data centers, populating them with hardware, purchasing additional software licenses, establishing, testing and maintaining fail over procedures is nontrivial. When you consider the overall health of the airline industry, it's not surprising that the extra tens of millions of dollars were not spent.

    It'd be interesting to know how many millions of dollars this will end up costing US Airways. I'll bet accepting the problem saves money over solving it. If you had a car worth $2000, you wouldn't spend $10000 to insure it. That's a rational decision.

  9. Last, and Dead Last by xkr · · Score: 3, Insightful

    I fly US Airways regularly. Last flight out was late taking off for no apparent reason. Our luggage did not make the connection in their own Hub. Neither did anybody else's. It took over an hour for the luggage clerk to process the long line. I counted over 500 keystrokes required per person. Staff didn't care at either airport. They would not put out luggage on the next plane in (another airlines, and they would have to pay a fee to that airline) so it was over a day to get out luggage. Two days, or three, unless we came back to the airport to pick it up. On the way home to SFO, it took over an hour for them to get out luggage onto the carousel. They had the nerve, over the PA system, to blame the passengers for having, "too much luggage," for the delay.

    Consumer Reports rated US Airways at the bottom of customer satisfaction.

    Planes fly. Southwest regularly makes last second changes, including flag stops (unscheduled) and re-using planes for "second runs."

    There was LOTS that US Airways could have done. First, they could have flown the planes if they wanted too. They planes had already been scheduled, so there were no questions of maintenance or fuel, or flight plans. Second, they could reimburse passengers for the delays. Third, they could have rescheduled some passenger.

    Then, of course, as said, there is simply no excuse for the IT to be down for that long, if at all. They had no (working) backup systems, either computers, paper, or people. That is the very definition of incompetent.

    I work in IT. As a guy said in my last meeting, “Anybody who designs in RAID 5 should be shot.” Duh.

    The fact is that the airlines management is incompetent. This is not an opinion. Simply too many facts. The board should completely clean house. When the questions comes up in the next board meeting of, “What to do?” the answer is, “Duh.”

    --
    I will create a sig when innovation restarts in the U.S.