Slashdot Mirror


Computer Glitch Friday Grounded US Airways Flights

mschaffer writes "A computer glitch Friday night snarled the travel plans of US Airways customers, as reports flooded in of flights grounded around the country." As someone stranded for several hours yesterday by this outage, "glitch" seems like quite a euphemism. With outgoing flights blocked, and new ones arriving full of passengers expecting to meet connections, the atmosphere got a little heated. Customers could see nice weather, and planes lined up outside, but "The System Is Down" trumps all. The E concourse at Charlotte (a US Airways hub) was packed full of customers ranging from livid (a handful) to merely angry (most) to calmly resigned — which means those of us with seats, snacks, and books or computers. It was disheartening to see how brittle is the infrastructure the airline employs; with the part of the system visible to airline employees down, customers thought they might get more information, or even rebooking, through the US Airways website. But that was down, too, and all the desk staff could do is shrug.

94 of 140 comments (clear)

  1. COBOL's fault, I'm sure! by Kensai7 · · Score: 4, Funny

    OK, let's count to three and blame COBOL! :p

    --
    "Sum Ergo Cogito"
    1. Re:COBOL's fault, I'm sure! by Anonymous Coward · · Score: 1

      Probably TPF running assembler, c and any number of Java connections

    2. Re:COBOL's fault, I'm sure! by rvw · · Score: 1

      Probably TPF running assembler, c and any number of Java connections

      TPF, what that? A Turbo Pascal Fuck?

  2. umm... by datapharmer · · Score: 2, Interesting

    that sucks. No backup paper system in place? Can't they just read what the tickets say such as flight and seat number? They know where the flights are going as most are routine. It seems they should have been able to get *some* flights in the air.

    --
    Get a web developer
    1. Re:umm... by linest · · Score: 3, Interesting

      You can't fly unless you can prove your aircraft has had all required maintenance done. There are also rules about the number of hours per day crew members are allowed to be in the air. I suspect these records could be printed and used if it were a planned outage but this wasn't.

    2. Re:umm... by DesScorp · · Score: 5, Informative

      I work in airport IT, so I'll describe what I see airline crews doing during trouble. If the system at one gate or terminal is down, yes, then they'll send the plane on it's way. This is called "boarding manually". They simply hand collect tickets, hand count bags, etc, and send the flight off. After they've gathered all of the info thats been collected manually, they'll send it to their local office or front desk and process it at working terminals that have a connection to airline systems. It's a pain, but do-able. But if EVERYBODY is down, then the whole thing grinds to a halt. If no one has any access to all the schedule info, weight and baggage, manifests, etc.... then it's simply impossible to board manually on a massive scale.

      --
      Life is hard, and the world is cruel
    3. Re:umm... by DerekLyons · · Score: 2, Informative

      No backup paper system in place?

      One of the reasons they went to computers in the first place is because paper systems could no longer handle the workload... And that was back in the 60's when air traffic volumes were a fraction of what they are today. I.E. having to maintain a duplicate paper system would actually slow things down likely without actually providing sufficient backup.
       

      Can't they just read what the tickets say such as flight and seat number?

      That isn't much help with getting the luggage on the appropriate aircraft. Nor does it help to inform what flights (that you're expecting passengers from) are on time or nearly so and will or will not effect the flight in question. (Let alone routing the luggage involved.) Not to mention the number of passengers and the weight of the luggage - something the pilot needs to know to operate the aircraft safely.
       

      They know where the flights are going as most are routine.

      There's a lot more information flowing through the system than just "plane A goes to destination B" and "butt X goes into seat Y". With the system down they don't even know when/where plane 'A' is in order to get butt 'X' onto it.
       

      It seems they should have been able to get *some* flights in the air.

      No offense, but that's because you don't even remotely understand the problem. (And seemingly can't even be bothered to try by asking questions rather than making statements.)

    4. Re:umm... by Anonymous Coward · · Score: 1

      they were faxing the necessary paperwork. Was supposed to pick up a friend @ SFO coming from Charlotte. They stated the systems were down and they had to fax all the pilot's info in order for the plane to take off.

    5. Re:umm... by FlyingGuy · · Score: 3, Interesting

      In 1960 most everyone was running Sabre

      Most everything was connected to hard lines that went back to the big main frame machines that ran it.

      If one terminal was down then it was most likely the terminal that had failed or possibly one of hundreds of hard lines back to the Main Frame

      Now days with everything being all cloudy good luck figuring out what might still be available. It could have been something as simple as the single bit of fiber serving that main concourse was damaged or a router someplace in the airport had failed or some cloud vendors routing had gone south or hell it might have been something as dramatic as what happened to Amazon a little bit ago.

      --
      Hey KID! Yeah you, get the fuck off my lawn!
    6. Re:umm... by Chaos+Incarnate · · Score: 2

      They didn't have Homeland Security wiggling their fingers and watching the airlines and passengers dance like marionettes.

      --
      Benford's Corollary to Clarke's Law: "Any technology distinguishable from magic is insufficiently advanced."
    7. Re:umm... by h4rr4r · · Score: 1

      It's not the thinking that is the problem it is the record keeping.

      How do you check that the plane has had the required maintenance?
      How do you check that the pilot has not been flying too long today?

      Not like they have massive filing cabinets at all the airports for this stuff.

    8. Re:umm... by Grishnakh · · Score: 2

      The number of passengers per day in 1940 was next to nothing, and today it's staggering.

      Similarly, back in 1940 or 1950, "long distance" telephone calls (remember those?) were routed manually, by operators at switchboards. Can you even imagine trying to go back to that, with the sheer number of "long distance" telephone calls going on these days?

      When you do things of great complexity in huge volume, you need to have computers do them. Sure, it's possible to do them manually if you already have an army of trained personnel at hand, and equipment and processes in place for them to use and follow. But it's not like you can just go back to that in an instant. Just like there are no more manual operator switchboards at telephone companies to fall back to, there's no way to fall back to handling flights manually.

      The problem is that many things that are automated simply aren't designed with sufficient redundancy and reliability. Some high-profile outages will fix that though.

    9. Re:umm... by Trilkin · · Score: 1

      That's just it, though. The scale has increased DRAMATICALLY. Back in the early days of commercial flight, it was generally very expensive and there were very few people able to take advantage of it plus the population of people more than a little nervous of sitting in big flying tin cans. It was, thus, easy to keep records when you only have a handful of planes and passengers with practically no security.

      --
      Nobody cares what the CAPTCHA for your post was.
    10. Re:umm... by jd · · Score: 1

      I know Slashdotters don't always have the best manners, but this isn't Kuro5hin. We still have standards.

      Whilst I agree that paper backup is probably out of the question, most computers are quite capable of handling multiple ethernet lines and most routers are capable of supporting hot standby configurations. Even cold standby is a 30 second failover. The same goes for backend servers - it doesn't take much to add a checkpoint/failover system (cold standby) and it's quite possible to configure most servers to support hot standby.

      Asteroid takes out a data center? Well, then you've probably got bigger issues, but co-locating across the country is Standard Practice for most instustries.

      This simply isn't about the problem. It's about whether the solution has been implemented. Nothing more.

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
    11. Re:umm... by egburr · · Score: 1

      Overall, I agree with what you said. Only one part didn't make any sense, though: "With the system down they don't even know when/where plane 'A' is in order to get butt 'X' onto it."

      Couldn't they just look out the window? Maybe that big thing outside the window that a whole bunch of people recently walked off of just happens to be the plane everyone at the gate is waiting to board?

      Yeah, just loading it up and taking off would make a huge mess of the paperwork, but don't tell me they can't find the plane.

      --

      Edward Burr
      Having a smoking section in a restaurant is like having a peeing section in a swimming pool.
    12. Re:umm... by digitig · · Score: 4, Insightful

      This simply isn't about the problem. It's about whether the solution has been implemented. Nothing more.

      And that is down to whether it is cost-effective to implement the solution. You will never be able to get the probability of failure down to zero. and the cost skyrockets the closer you get to zero. How often do outages like this happen, and how much would it cost to prevent them at every airport worldwide? And to prevent every other conceivable scenario? Yes, it could have been prevented, and lots of other possible outages that didn't happen could have been prevented, but the cost of air tickets would be prohibitive.

      --
      Quidnam Latine loqui modo coepi?
    13. Re:umm... by Deadstick · · Score: 3, Interesting

      Rent an amusing sf film made in 1953 called "The Magnetic Monster" and watch an airport official tell Richard Carlson "We can't search ALL the flights. This is Los Angeles International; we have over a dozen departures a day!"

      rj

    14. Re:umm... by timothy · · Score: 1

      One of the things I saw yesterday was gate switches; the desk staff don't know until they get the (apparently very sporadic) updates about things like that, so people are often directed to distant gates rather than their originally scheduled ones. And when it happened yesterday, I saw two different flights (to different cities!) both being sent to the same gate, and I'm pretty sure they weren't both right. Glad I just had to wait, and it was my final leg.

      timothy

      --
      jrnl: http://tinyurl.com/c2l8yr / foes: http://tinyurl.com/ckjno5
    15. Re:umm... by X0563511 · · Score: 1

      Yay fax! A technology older than voice telephones.

      Why the fuck are we still using faxes?

      --
      For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
    16. Re:umm... by linest · · Score: 1

      Why the fuck are we still using faxes?

      Because the computers are down?

    17. Re:umm... by DerekLyons · · Score: 1

      I know Slashdotters don't always have the best manners, but this isn't Kuro5hin. We still have standards.

      They vary wildly between 'unreasonably high' and 'ludicrously amusing', but yes, Slashdot has standards. (And very occasionally, the have some relevance to the real world.)
       

      Asteroid takes out a data center? Well, then you've probably got bigger issues, but co-locating across the country is Standard Practice for most instustries.

      If 'most' industries had a system even half as complex - you'd have a point. But 'most' industries don't dynamically track dozens of airports, hundreds of aircraft, and tens of thousands of passengers (and their luggage) in real time. Worse yet, not only in this information dynamic, its also tightly interrelated. (That plane I'm meeting in Dallas after taking of from Sea-Tac this evening is the same plane that flew from LA to Chicago this morning, and then flew Chicago to Dallas.)
       
      This isn't just a simple matter of backing up data and providing redundant communications links... It's also a matter of keeping the hot spare updated and parallel to the active string at about the triple nine level. This is both very expensive and very hard. (And requires something much more than the standard PC and server mentality the average Slashdotter deals with.)
       

      This simply isn't about the problem. It's about whether the solution has been implemented. Nothing more.

      As with the poster to whom I replied, you don't even understand the problem. This results in facile and inappropriate 'solutions' that don't even begin to address the issues.

    18. Re:umm... by DerekLyons · · Score: 1

      Maybe that big thing outside the window that a whole bunch of people recently walked off of just happens to be the plane everyone at the gate is waiting to board?
       
      Yeah, just loading it up and taking off would make a huge mess of the paperwork, but don't tell me they can't find the plane.

      Assuming the plane is at the gate - what about an hour before when it's still a couple of hundred miles away? What happens if it's delayed, or never even takes off? How do they route the luggage since it doesn't have the tags that they can't print because the system is down?
       
      As I said before, not only do you not understand the problem, you can't be bothered to even try.

    19. Re:umm... by digitig · · Score: 1

      It wasn't just a simple power outage at a data centre, though, as far as I can see. And even if it was, it's still a cost-benefit analysis to decide whether it's worth doing anything about it. If you make all of your systems multiply redundant with no regard for the cost-benefit then you are not going to stay in business for long.

      --
      Quidnam Latine loqui modo coepi?
    20. Re:umm... by jd · · Score: 1

      Having worked in places with hundreds of remote offices, for that matter having worked at CERN on data collection for nuclear accelerators, I think I might, just might, have an idea of what it takes to keep large numbers of systems in sync over continental distances.

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
    21. Re:umm... by jd · · Score: 1

      It's a huge transactional database, yes. What's your point?

      When building a fault-tolerant system you've two choices:

      a) Build the system such that you have N databases (not 1) with transactions sent to any of them replicated across the others. By doing transactions in bulk over some period (say every 5 mins), everything within that 5 min window will always be in the correct order on all instances of the system. By having the kiosks retain the past 5 mins of transactions and the electronic address of each server, if one goes down it can round-robin to the next and dump the buffer into it.

      b) Use NACK-oriented reliable multicast and have each transaction automatically forwarded to all N servers at the same time.

      You use leased lines and MPLS to create the virtual circuits needed for this. One fully reliable, trans-continental system. Because each kiosk "sees" one server (even though it may change at any given time) and that server is always current, it "sees" a single master system even though no master system exists.

      Look, this is stuff I know you youngsters have a hard time understanding, but my generation had already solved these problems before half of the current Slashdot population was born. Now gerroff my lawn!

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
    22. Re:umm... by FlyingGuy · · Score: 1

      Not to mention hoping the pilot gets the weight & balance correct, not to mention fuel consumption ( which none of them have had to calculate in years ). Most times the plane datalink has the correct information but I have seen a commercial jet parked at the gate because the feed from the computer was down.

      Making assumptions based on the standard FAA passenger w/standard baggage is tricky business not to mention dangerous since that has killed more then a few passengers and crew especially stops at regional airports where some get off some get on and off we go! Out of range CG kills people.

      --
      Hey KID! Yeah you, get the fuck off my lawn!
    23. Re:umm... by mcmonkey · · Score: 1

      Fine, so replace the question with 1950, or 1940. There have been airlines for a long time and I'm sure they were able to board planes before they had computers.

      Remember the recent story on the Harvard entrance exam from thousands of years ago? I do, and I don't recall too many people bragging about how well they would have done on the Latin and Greek sections. Does that mean thousands of years ago people were geniuses, and the modern /. poster is an idiot?

      No. It just means people don't routinely study Latin and Greek anymore.

      Likewise, whatever system the airlines were using pre-networked computers may very well work today to keep things moving. Except no one working the terminal remembers that system.

      So are you proposing the hundreds of people working the terminal reinvent that pre-computer system? Each one on their own? Or shall they all abandon the stranded passengers, and come together to devise a replacement for the down systems?

      Remember, those folks who ran things without computers, they still had training and a system provided for them. They weren't just making things up on the fly.

  3. Shit happens by Osgeld · · Score: 1

    sometimes its a pain

    1. Re:Shit happens by Anonymous Coward · · Score: 1

      I'd just like to ask timothy to revisit this post after he's had some months or years. Being stranded for several hours is pretty frickin' minor as far as bad things that can happen when flying or in life in general; calling it a 'glitch' isn't much of a euphemism, although I might call it a 'major glitch'. Bad things seem much worse when they happen to you, but when you're an adult you're supposed to be able to get some perspective on it.

    2. Re:Shit happens by timothy · · Score: 1

      Yep -- I was fine.

      Like I said: snack, seat, plenty to read ... my Slashdot work for the day was done, too, so no one else was being much inconvenienced by my travel delays in particular. (It's the other people, with connections etc, who had bigger hassles.)

      This is so far from the worst thing that could happen even among modern travel disruptions that I hope you take my account with the same viewpoint I had: it's a 21st century problem / first-world problem, and that's the best kind of problem to have. Once I was stuck in San Antonio for 3 days because of ice storms ... which was actually pretty fun, since I wasn't missing a friend's last moments, an organ transplant, etc.

      The worst aspect for those people who didn't have cliff bars, gum, etc. is the uncertainty -- don't know if you have time to go snag an airport-priced sandwich, because you might miss an important announcement.

      And my favorite on this sort of problem remains this: http://www.youtube.com/watch?v=8r1CZTLk-Gk

      --
      jrnl: http://tinyurl.com/c2l8yr / foes: http://tinyurl.com/ckjno5
  4. Us air by jbrodkin · · Score: 1

    Ive had more problems with us air and united than any other airline. Theyre incompetent

    1. Re:Us air by oneiros27 · · Score: 1

      And that would match with J.D. Power's customer satsfaction reports for 2011:

              http://www.jdpower.com/travel/ratings/airline-ratings/traditional/

      Although, when you compare the "Low Cost" airlines, there's a few others that had similar bad ratings:

              http://www.jdpower.com/travel/ratings/airline-ratings/low-cost/

      --
      Build it, and they will come^Hplain.
    2. Re:Us air by nwf · · Score: 1

      And it would match Consumer Reports who rated usair the worst of any domestic carrier. I'd link, but they keep almost everything behind a paywall. Usair used to be really good, until they merges with America West. United is second to last and that's who I'm waiting for now at the airport. Only 75 min late currently.

      Since united and continental are merging, I'll bet the combined airline will be far worse than either alone.

      --
      I don't know, but it works for me.
    3. Re:Us air by ShakaUVM · · Score: 1

      Yeah, that's a good point. If I lived in Atlanta, I'd probably fly Delta more often.

      Never had complaints with United. I fly a couple times a month. Well - we had a hydraulic issue once, they deplaned us and gave us free food and drink. Pretty reasonable, really.

      United scores points in my book for EcobomyPlus service. Not many others offer the legroom of first class for free(ish).

      My hatred for USAir is incomparable, though.

    4. Re:Us air by Drathos · · Score: 1

      I've never had any major problems with USAirways (a few delays here and there, but only one more than 30 mins). I actually flew through CLT yesterday, but I guess I got out before the problems started.

      Delta, on the other hand, is a constant problem. Left stranded in ATL twice, many delays, and terrible customer serice. Maybe if they didn't overload ATL, it wouldn't be so bad (that doesn't help the customer service issues, but it might with the others). I avoid them whenever possible.

      --
      End of line..
  5. Backups? by fysdt · · Score: 1

    Why isn't there a backup available in case a glitch occurs?

    1. Re:Backups? by RdeCourtney · · Score: 2

      Because they've calculated the customer apathy and money lost is less than implementing backup procedures. Remember it's all about $.

      --
      Insert signature here...
    2. Re:Backups? by CohibaVancouver · · Score: 1

      Why isn't there a backup available in case a glitch occurs?

      It's called "risk management." Let's say a backup system would cost $150M over 20 years, and the current system is calculated to fail every seven years, at a cost per failure of $30M (cost in terms of lost business / OT / brand damage etc.). Running without the backup system you're many tens-of-millions of dollars ahead in the game. These figures are just made up, but these sorts of calculations go on all the time in many different industries.

    3. Re:Backups? by linest · · Score: 2

      Because they've calculated the customer apathy and money lost is less than implementing backup procedures. Remember it's all about $.

      I'm not crazy about the way that's phrased, but you are essentially correct. Establishing backup data centers, populating them with hardware, purchasing additional software licenses, establishing, testing and maintaining fail over procedures is nontrivial. When you consider the overall health of the airline industry, it's not surprising that the extra tens of millions of dollars were not spent.

      It'd be interesting to know how many millions of dollars this will end up costing US Airways. I'll bet accepting the problem saves money over solving it. If you had a car worth $2000, you wouldn't spend $10000 to insure it. That's a rational decision.

    4. Re:Backups? by jd · · Score: 1

      That depends on how you define cost. The instantaneous cost is one part, but only a part. Even so, that cost isn't just frustrated customers. It's parking costs for the aircraft, staff wages, any technician overtime needed, costs due to food spoilage, loss of in-flight sales, etc. Delayed costs also matter. There's any loss of future custom to consider, since that is also a cost to the company. There's any increase in insurance costs for them as a result of any successful claims. It may well impact the airline's ability to purchase space at an airport or purchase a specific route. All these things are costs.

      The problem is that many aren't quantifiable - too many unknowns - so an airline is incapable of knowing if a backup system is cheaper or not.

      Oh, as for additional software licenses, many enterprise-level software vendors support floating licenses. So if you're doing cold standby (a whopping 30 seconds of outage), no additional licenses are needed.

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
    5. Re:Backups? by jd · · Score: 1

      Fail over the TCP/IP connections (hot standby) or recover from a checkpoint (cold standby), re-sync with events and then continue. Why do you ask?

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
    6. Re:Backups? by Sponge+Bath · · Score: 1

      There was a backup plan in the suit pocket of the CIO, but it was casual Friday, so everyone was, you know, wearing a Hawaiian shirt.

    7. Re:Backups? by digitig · · Score: 1

      The problem is that many aren't quantifiable - too many unknowns - so an airline is incapable of knowing if a backup system is cheaper or not.

      But the airline still has to make the call. Pay for a backup for system x or take the hit if it fails. And they can only sensibly make that call if they can make an estimate of how much the hit will cost and how often it's going to happen. And don't forget: multiply redundant systems can fail, too. I've seen a power system based on main and standby UPSs fed by different power supply companies, backed up by main and standby generator sets, each of which comprising main and two standby generators, go down completely at an air traffic control centre (due to one maintenance issue and one design error). Just how much do you want the airlines and airport to spend to make sure something like this doesn't happen? How much are you willing to pay for your ticket?

      --
      Quidnam Latine loqui modo coepi?
    8. Re:Backups? by linest · · Score: 1

      The problem is that many aren't quantifiable - too many unknowns - so an airline is incapable of knowing if a backup system is cheaper or not.

      I like that observation a lot. It could be carried further. You're talking about actual risks. Real decisions are made based upon perceived risks that sometimes consist of little more than assumptions. Especially once you get out of the IT realm and need something paid for. Since these are IT risks and they need to be communicated to non-IT people, there is a challenge there. It's not easy.

      Your comment about software licenses being free for disaster recovery, on the other hand, seemed a bit too off hand. I believe the issues are the databases for crew scheduling and aircraft maintenance. That's the stuff that'll keep you on the ground. To my knowledge, there is no dominant application for aircraft maintenance packages. If that data is stored in an Oracle database, you're going to pay big bucks for DR licenses. On the other hand, if we assume that US airways is using a Jeppesen package for crew scheduling (I'd bet a small amount of money on this), then it relies on a 10 year old version of Informix. Disaster recovery for the database server software would (in my experience) be free. IBM's OK that way.

  6. Very interesting by jhoegl · · Score: 1

    I dont know what the "outage" was, but it seems redundancy is an afterthought with US Airways.
    So, I can assume the following
    There is no redundancy(zing)
    There is no Recovery plan
    There is no DR plan
    There is no SoP on releases
    There is no SoP on testing
    There is no SoP on handling outages
    Bravo! Now, did those new TSA rules go into effect yet? Can US Airways be fined into oblivion because of this?

    Also, what are the extra fees US Airways charged its passengers for having to handle their complaints and angry faces? $100 per complaint?

    1. Re:Very interesting by brusk · · Score: 3, Insightful

      Would you buy tickets that cost $25 than anywhere else more because the airline advertised redundancy in its IT systems? People choose flights based on price (secondarily, on frequent flier plans, etc.), but how is a consumer supposed to choose based on factors like this, except in the most general terms (on-time percentage)? The airline management knows this, and would be silly to invest too much in things that will raise costs without enabling them to increase revenues.

      --
      .sig withheld by request
    2. Re:Very interesting by jd · · Score: 1

      Airlines with outstanding reputation for timliness and customer service probably could charge $25-$50 more per ticket and have the customers grateful for it. The problem is that it takes decades to build that kind of rep but mere seconds to destroy it. Much easier to pretend to cater for the unwashed masses because that means guaranteed profit now rather than a higher but riskier long-tern profit due to good, competent service.

      It's the way Microsoft, T-Mobile, Comcast and talk-show hosts have bilked people for years.

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
    3. Re:Very interesting by Idbar · · Score: 1

      I like to check flightstats and check if the flights I'm picking have bad statistics about being late most of the time. In that sense I try to avoid stopping in Denver and sometimes Las Vegas.

      I know stuff happens, but if I can minimize the amount of things that can go wrong, I'm willing to pay a bit more.

    4. Re:Very interesting by ShakaUVM · · Score: 1

      People that would pay $25 for a better flying service are NOT flying US Airways anyway. It has the worst customer service of all the legacy carriers.

    5. Re:Very interesting by digitig · · Score: 1

      Airlines with outstanding reputation for timliness and customer service probably could charge $25-$50 more per ticket and have the customers grateful for it.

      Yes, and other customers will choose the airline that charges $25-$50 less per ticket to save the money, then gripe when they get on the wrong side of the inferior reliability.

      --
      Quidnam Latine loqui modo coepi?
    6. Re:Very interesting by sgtrock · · Score: 1

      It's the way Microsoft, T-Mobile, Comcast and talk-show hosts have bilked people for years.

      You're going to single out T-Mobile??? The one carrier in the U.S. that actually had an HTC Android phone as soon as it was released? The one carrier in the U.S. that actually seems to give a damn about not loading up their phones with all the garbage the others do? The carrier in the U.S. who doesn't make life hell if you want to use a phone that you bought some place else?

      Granted, their coverage for data outside major metropolitan areas can be spotty, especially west of the Mississippi. Their coverage maps make no secret of that fact. It's a LONG leap not being satisfied with their coverage to putting them in the same category as Microsoft and Comcast. If any of the U.S. carriers belong there, it's the big three; AT&T, Sprint, and Verizon!

    7. Re:Very interesting by drsmithy · · Score: 1

      Airlines with outstanding reputation for timliness and customer service probably could charge $25-$50 more per ticket and have the customers grateful for it.

      They couldn't, because the vast majority of their customers don't fly frequently enough to care, and/or purchase solely based on price.

      People who do fly frequently enough to care, are generally protected and/or bought off by their frequent flyer status (frequent flyer programs in the US are _very_ generous).

    8. Re:Very interesting by 19061969 · · Score: 1

      I pay a lot more than $25 extra to fly with airlines that don't keep pissing me and my family around with seat re-allocations, and I've often avoided poorer quality airlines offering cheap prices).

      Price is a factor but it's not the only one. Availability is also a factor as are length of stop-overs, chance of getting bumped up, total flight length, where I have to transit and so on.

      Proviso: I rarely fly in the US and my airlines of choice are Singapore, Emirates and Air NZ (the latter of which are often way too expensive so I prefer to go SK or EK which are so far ahead of US carriers it's not real. It's strange considering the US usually has excellent levels of customer service elsewhere why it's so poor for air travel. I guess I'm lucky in that I can often avoid US airlines if traveling to the US.

      --
      bang goes my karma... again...
  7. Computer Glitch Friday? by QuasiSteve · · Score: 1

    I know IT fully embraced Patch Tuesday leaving us with up to a month's worth of accumulated crud, but now they've gone too far!

    1. Re:Computer Glitch Friday? by Beelzebud · · Score: 1

      I read it exactly the same way. I guess using "A" and "on" was too much extra typing. :D

    2. Re:Computer Glitch Friday? by jd · · Score: 1

      Maybe the computer glitch is an artificial intelligence called Friday. It then called up the airports and ordered the planes not to take off.

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
  8. Wait, what? by Anonymous Coward · · Score: 2

    FTFA: "The Tempe, Ariz-based carrier cited a power outage near one of the airline's data centers in Phoenix as a possible cause."

    A POWER OUTAGE?! So, no UPSes, no generators, and no multiple utilities at a main data center for a major company? Come on now...

  9. What?! by Anonymous Coward · · Score: 1, Insightful

    I love it when people melt down and scream and yell. Fucking christ. Shit happens. Shut your mouth, go over to the airport Starbucks and buy yourself an overpriced airport coffee and calm down. You're not helping. You're not the center of the universe, you twit; in fact, they're probably not even going to even up the scales.

    (I'm one of those in the calm catagory when it comes to Emergencies, or (more likely) "emergencies.")

    1. Re:What?! by egburr · · Score: 1

      A cup full of caffeine and sugar is supposed to help you calm down?

      I used to carry a couple extra books in my carry-on bag, as I have seldom had a flight even come close to being on time (except for connecting flights which are almost always on time no matter how late my initial flight is).More recently, I just make sure my phone charger is with me, so I can keep my battery topped off as I read ebooks or play games on my phone instead. My only real complaints are that I have yet to find an airport with even slightly comfortable seating and that there is never any place at all to get away from all the noise of the multiple TVs tuned to multiple stations all at full volume and the incessant security announcements just in case there is someone in the airport who hasn't flown in the past 20 years.

      --

      Edward Burr
      Having a smoking section in a restaurant is like having a peeing section in a swimming pool.
    2. Re:What?! by KingAlanI · · Score: 1

      Yeah, I also thought that Starbucks wasn't the best example.
      Last I I flew, it was for once it was good that I tend to overpack, as that in part included extra reading material that I hadn't gotten to the rest of the trip.

      --
      I listen to both RIAA and non-RIAA stuff if I like the music, tangential business/politics nonwithstanding.
    3. Re:What?! by Abstrackt · · Score: 1

      My only real complaints are that I have yet to find an airport with even slightly comfortable seating and that there is never any place at all to get away from all the noise of the multiple TVs tuned to multiple stations all at full volume and the incessant security announcements just in case there is someone in the airport who hasn't flown in the past 20 years.

      HHGTTG actually got this one right: bring a towel. You can use it as a cushion on an uncomfortable seat or even as a pillow if you want to nap on the floor. Just make sure it's a thinner towel, ideally a travel one, so it doesn't take up too much space in your bag/suitcase. As for the noise, IEMs are a godsend. Once my music or ocean sounds are on I don't hear anyone or anything else, perfect for those six-plus hour stopovers.

      --
      They say a little knowledge is a dangerous thing, but it's not one half so bad as a lot of ignorance. - Terry Pratchett
  10. United was also FUBAR yesterday... by neurocutie · · Score: 1

    I specifically chose United over USAir for travel yesterday as I've had the most trouble with them. However United was also in poor shape yesterday. It was termed 'operational delays', with two hour delays across the board. Calls into United faced 25-30min wait times. And many overbooked flights.

    Seems the whole industry is going down the tubes... and decreased competition from these mega-mergers are not helping.

  11. Could you tell the difference? by Cutriss · · Score: 4, Interesting

    As I am now located in proximity to an airport with a US Airways "service focus" and have had the "pleasure" of flying with them several times, I have to ask - how would you be able to tell the difference? Every time I've been in a US Airways terminal, there's always a significant number of non-weather-related delays and cancellations (compared to the other airlines' monitors). My wife and I have independently had three separate incidents this year where we were 4th and inches from having to stay overnight at an airport due to cancellations/late planes/overbooked crew/etc. In two of those cases, I had flights where we took off at the 2'55" mark, just shy of the three hour requirement to return to gate and let everyone off. The cynic in me suspects that US Airways is actually using that three hour window to plan its flights.

    It's an abhorrent mess, and when I see the US Airways CEO defending against his last place customer service ranking, I have to wonder just how much denial one management team can stand.

    --
    "Mod, mod, mod...and another troll bites the dust."
    1. Re:Could you tell the difference? by jittles · · Score: 1

      See I have the exact opposite experience with US Air. Free upgrades to first class, and have not experienced any more delays than any other airlines. My worst experiences have been with Delta and AA.

  12. Whatever did we do... by Anonymous Coward · · Score: 2, Insightful

    ...before computers came along and made our lives so much -easier-?

    1. Re:Whatever did we do... by jd · · Score: 1

      I dunno. The R100 was a flying hotel and a fleet of those would have been quite capable of carrying the same number of passengers modern airlines could.

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
    2. Re:Whatever did we do... by 246o1 · · Score: 3, Insightful

      ...before computers came along and made our lives so much -easier-?

      Not fly anywhere, because it was too expensive.

      --
      Although the moon is smaller than the earth, it is farther away.
    3. Re:Whatever did we do... by blair1q · · Score: 1

      they weren't supposed to make your life easier.

      they were supposed to make your ticket price leave more profit in the airline's pocket.

      all those people who couldn't get anywhere? just how many got their money back?

      the system is working exactly as designed, even if the passengers don't get to feel all special about it.

    4. Re:Whatever did we do... by DerekLyons · · Score: 2

      Back in the 60's when airlines started computerizing, air traffic volume was a fraction of what it is today. You'd have to be nearly fifty to have even been alive at a time when computers weren't starting to make our lives easier in a variety of ways. ("Computers" !=" PC's".)
       
      I remember trying to make airline reservations back before the web. You couldn't pay me enough to go back to those days. (Unless you could also give me my twenty year old body as well.)

  13. I think Louis CK sums it up well by Time_Ngler · · Score: 1
  14. Last, and Dead Last by xkr · · Score: 3, Insightful

    I fly US Airways regularly. Last flight out was late taking off for no apparent reason. Our luggage did not make the connection in their own Hub. Neither did anybody else's. It took over an hour for the luggage clerk to process the long line. I counted over 500 keystrokes required per person. Staff didn't care at either airport. They would not put out luggage on the next plane in (another airlines, and they would have to pay a fee to that airline) so it was over a day to get out luggage. Two days, or three, unless we came back to the airport to pick it up. On the way home to SFO, it took over an hour for them to get out luggage onto the carousel. They had the nerve, over the PA system, to blame the passengers for having, "too much luggage," for the delay.

    Consumer Reports rated US Airways at the bottom of customer satisfaction.

    Planes fly. Southwest regularly makes last second changes, including flag stops (unscheduled) and re-using planes for "second runs."

    There was LOTS that US Airways could have done. First, they could have flown the planes if they wanted too. They planes had already been scheduled, so there were no questions of maintenance or fuel, or flight plans. Second, they could reimburse passengers for the delays. Third, they could have rescheduled some passenger.

    Then, of course, as said, there is simply no excuse for the IT to be down for that long, if at all. They had no (working) backup systems, either computers, paper, or people. That is the very definition of incompetent.

    I work in IT. As a guy said in my last meeting, “Anybody who designs in RAID 5 should be shot.” Duh.

    The fact is that the airlines management is incompetent. This is not an opinion. Simply too many facts. The board should completely clean house. When the questions comes up in the next board meeting of, “What to do?” the answer is, “Duh.”

    --
    I will create a sig when innovation restarts in the U.S.
    1. Re:Last, and Dead Last by T-Bucket · · Score: 1

      You are obviously clueless as to how an airline is run.

      They could not have just "flown the planes if they wanted to". Despite scheduling the flights, EVERYTHING is handled by computers now. Those flight plans you mentioned? Yep, filed by a dispatcher USING A COMPUTER. The performance calculations that determine how much fuel that flight plan will require? Yep, computer.

      Add to this the fact that the computers also control gate assignment, weight and balance, baggage routing, etc etc. There is NO WAY a modern airline can run their entire operation without computers. PERIOD. It's just not possible.

      (And yes, I do fly for an airline.)

    2. Re:Last, and Dead Last by Xacid · · Score: 1

      “Anybody who designs in RAID 5 should be shot."

      This runs counter to everything I've been taught - so perhaps you have some real world advice to support this? Merely curious.

    3. Re:Last, and Dead Last by Idbar · · Score: 1

      Even worse, they have managed now to blame anything and everything on weather so they don't have to reimburse people. Such that when they should have a properly deicer working (but they don't), they'd blame it on the cold weather.

    4. Re:Last, and Dead Last by IQgryn · · Score: 1

      RAID 5 is good at losing a second disk soon after the first, even with a hot spare (which means you lose the array). I think the GP meant something like "RAID 6 should be the bare minimum", or perhaps "any level of RAID is not as good as having an independent backup system".

      But I'd like to know what they actually meant, too.

    5. Re:Last, and Dead Last by swalve · · Score: 1

      Yeah, OMG, RAID5 doesn't do things it wasn't designed for! Use proper hardware (Proliant/Dell) and it works just fine for what it is. A cheap way to drastically improve uptime. RAID isn't about anything more than improving uptime. And I've NEVER seen double disk failures in systems using said hardware and that had someone walk past the machine once a week or so looking for orange lights. ONCE I saw an array freak out because during a rebuild, a bad spot was discovered on another hard drive. But it was a huge array and really should have been raid6, and this would have been discovered if they were doing regular raid integrity checks.

    6. Re:Last, and Dead Last by DerekLyons · · Score: 1

      There was LOTS that US Airways could have done. First, they could have flown the planes if they wanted too. They planes had already been scheduled, so there were no questions of maintenance or fuel, or flight plans.

      If it were a static problem, you'd have a point. But it's a dynamic problem involving dozens of airports, hundreds of aircraft, and tens of thousands of passengers and their luggage. Not to mention that flight plans are filed immediately before departure... so, no system, no flight plan. Not to mention that fuel calculations are performed shortly before departure, again - no system, no fuel calculations, unsafe as hell to fly. Not to mention that maintenance occurs constantly, so... well, hopefully by now you get the point.
       

      Then, of course, as said, there is simply no excuse for the IT to be down for that long, if at all. They had no (working) backup systems, either computers, paper, or people. That is the very definition of incompetent.

      Proof positive that you can work in IT and still be an idiot.

    7. Re:Last, and Dead Last by lanner · · Score: 1

      Yea, that's pretty much what he meant.

      The reality is that an entire flight scheduling system like the one that US Airways uses could probably be replaced by $50K worth of junk off of dell.com. The software has to be written custom, but this isn't computational proteomics here. A couple of SF bay goons could do this in six months.

      For this kind of a small-scale implementation, you should have at least three separate data centers across the world/continent, which duplicate the information with automatic terminal failover to the nearest operating master. You would not just want the array to be RAID, but duplicate arrays, duplicate servers, duplicate network infrastructure, duplicate entire systems three times over... and it would still cost 1/3rd or less of whatever money they are currently pouring into the black hole of incompetency that they have right now.

    8. Re:Last, and Dead Last by adri · · Score: 1

      Wow, and that likely completely underestimates the scale of this kind of project.

      Chances are there's lots of systems all tied together, feeding data in and out. There may not be a "small scale implementation". This may be a pain in the ass to integrate into lots of these inter-operating system.

      So maybe the system-as-a-whole is flawed, sure, but redesigning that to be less of a clusterfuck is likely not within the scope of your "$50k and 6 months."

      Then there's keeping everything in sync. You end up having lots of changes coming through at lots of times from lots of places. If you have backup datacentres, they also have to have copies of this data, kept in real time. If you flip to the backup, the master can't retake being master until it's brought into sync.

      If the software versions are out of date and they behave slightly differently, then you end up with fantastic and hilarious subtle issues. For example, if the software makes slightly different decisions about "stuff", then when you flip to the backup system, a whole lot of small, subtle changes in selection could snowball into much larger-scale problems down the track. Since a lot of "engineered" solutions go through periods of "design", "implementation", 10: "shit it didn't work as intended - take real life case A, B, C and implement workarounds", "workaround", "GOTO 10", you may find that these kinds of subtle issues are never really totally understood. They're worked around. Who knows what the problems are. So you flip to a backup system that has an impact on real, tangible things (such as say, luggage routing), a small change in decision making can go a freaking long way in affecting the physical world around you - and once you've started down the path of a physical-world clusterfuck, you don't always get to simply fix it with a software patch.

      Hm, what else can I think of off the top of my head. Oh right - what if the crash is because of the software + database contents? What if the crash is because the system overloaded because of some periodic job say, sweeping the database clean, or recalculating better routing rules. Chances are your backup sites will also fail - they're running the same hardware and software, with the same data? So you then say "run the backup software with an X minute delay, so if issues creep up in the primary we can stop the backups from failing the same way." The larger X is, the greater your chance of identifying problems - but the more useless it becomes when flipping over to become master. Stuff that has already been calculated by the (now failed) master hasn't been processed by the slave. So you have to integrate that data in, or design the system to cancel out the already-calculated stuff and re-issue things. Not so good if you have to cancel some real world event (eg fueling a plane) if the calculated values differ. So you choose to run primary and backups on different software versions, or revisions, or heck even independently designed systems. See paragraph 2 for the potential hilarity.

      This is why it's more difficult than $50k of junk and 6 months of goon time.

  15. Suprised? by Anonymous Coward · · Score: 1

    Given a choice between two competing airlines flying to the same place, the vast majority of passengers will book based solely on published cost.

    Leaving aside for the moment the question of hidden fees, that means that the airlines have no choice but to trim every possible cost to be competitive. I'm not in the airline industry, so I'm guessing here. I suspect that means no extra flight crews on standby for unexpected events, no extra gate crew coverage, the absolute minimum of phone lines to handle problems, etc.

    I recall flying 30 years ago, and there were airport staff sometimes standing around with nothing to do. But when things got weird, the resolution was much easier because of the flex built into the system. That was when consumers had brand loyalty (perhaps because they had no tool other than the phone to compare prices).

    On a recent trip there was a major weather event, and our plane ended up in a comedy of errors which would have been funny except for the fact that I was stuck in my seat for 12 hours. In the end, we were at an airport with no company terminals to dock at. The city run airport transportation staff had left for the night, so we couldn't be driven from the plane. According to the airport rules there, we couldn't use another company terminal unless there was an emergency. With the prospect of his passengers spending another 8 hours on the plane, the captain declared a "medical emergency" so we were allowed to deplane at the nearest empty gate. I will forever be grateful for that pilot, and the balls he had to do that, knowing that it might impact his career.

    The next morning when I tried to resolve my issue with the airline, every call to the customer service line was not met with "we regret that you will be on hold for 2 hours", but instead "we regret that we are not able to answer your call at this time", and then a dial tone. The web site offered no help either, claiming that due to system issues, they were unable to handle the volume of information requests. I ended up booking another flight on my own dime (well... the company's dime anyway) with another airline.

    Would I be willing to pay twice the cost for an airline flight to make this kind of crap go away? Sure. But I suspect I'm one of a few. I bet the vast majority of air travelers only go once in a while, and tend to forget what carrier they used last time, no matter how bad the service was.

  16. CAos by Etraud · · Score: 1

    Closer to Caos... lol

  17. Re:No computer glitches by jd · · Score: 1

    Good software is fault-tolerant. Fault-tolerant software DOES have a backup strategy if the programmer screws up. In the modern world, standards for software have fallen, not risen. If they had risen, virtually all software would be fault-tolerant and this kind of problem would not exist.

    --
    It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
  18. Re:No computer glitches by fast+turtle · · Score: 1

    who said it was software. What about some idiot with a backhoe that cuts a cable?

    What I don't understand is why in hell a major hub airport doesn't have something like a system that's setup to cache this data? To my mind, this would actually help in the case some idiot with a backhoe does cut a com line or they get hit with some disaster like a tornado, huricane or god forbid an earthquake taking out large swathes of infrastructure.

    --
    Mod me up/Mod me down: I wont frown as I've no crown
  19. Re:Snarled? by dtmos · · Score: 1
  20. Weather by T-Bone-T · · Score: 1

    I love the weather comments. They show that people just don't think on a large scale. I work at Dallas Love Field and I went home an hour late last night because of bad weather in the morning on the other side of the country. Most people would not realize that.

  21. No, they couldn't. by raehl · · Score: 1

    Couldn't they just look out the window? Maybe that big thing outside the window that a whole bunch of people recently walked off of just happens to be the plane everyone at the gate is waiting to board?

    Maybe, but probably not.

    If you land a plane at an airport, and go and park at a gate, what are the odds that the people waiting for a flight at that gate are supposed to be on the same plane that you just parked there?

    Even if you get lucky, how is anyone supposed to even know what gate they are supposed to be at?

    1. Re:No, they couldn't. by egburr · · Score: 1

      "If you land a plane at an airport, and go and park at a gate, what are the odds that the people waiting for a flight at that gate are supposed to be on the same plane that you just parked there?"

      Pretty good odds, actually. Every time I've been waiting at the gate when a plane arrived and disgorged passengers, that was always the same plane that we eventually got on. Except for one time they had to replace it due to mechanical problems.

      It doesn't make any sense to pull up to one gate, empty the plane, and then move it to another gate for a new load.

      --

      Edward Burr
      Having a smoking section in a restaurant is like having a peeing section in a swimming pool.
    2. Re:No, they couldn't. by egburr · · Score: 1

      "Even if you get lucky, how is anyone supposed to even know what gate they are supposed to be at?"

      It's printed on my ticket. Except for the rare last minute gate change, it's been pretty accurate.

      With the computers down, it can't be perfect, but for *most* of the flights, the available data is still good.

      --

      Edward Burr
      Having a smoking section in a restaurant is like having a peeing section in a swimming pool.
  22. "There is another system" by WaffleMonster · · Score: 1

    The word "backup" is often confused with "practice" ... backups who needs backups?

  23. United out of Chicago, eh? by KingAlanI · · Score: 1

    I fly very rarely; last time I did was August 2010 out of Chicago (O'Hare) on United (actually a United-branded regional carrier, but it went directly back to my hometown.) Get up early to make sure I'm to the airport on time, and the flight ends up delayed several hours.

    Insufficient sample size, I know...

    * my schedule had allowed me to cheap out and take Amtrak _into_ Chicago, which went by with only relatively minor delats.

    --
    I listen to both RIAA and non-RIAA stuff if I like the music, tangential business/politics nonwithstanding.
  24. Ah, like in the dutch rail network by SmallFurryCreature · · Score: 1

    Utrecht is a central city in Holland where pretty much all the railway lines intersect for no smart reason.

    So, if there is an issue at Utrecht like a fire alarm at the control center, there better be a backup or all train travel in Holland is seriously affected.

    Luckily the backup control center is there.... right there... in the same building... small building... affected by the same fire alarm...

    But hey, lets not immidiately order a load of busses to deal with stranded passengers, people have become so used to the troubles that they will no doubt fix it themselves, yet again.

    You can't run complex operations on a shoe-string budget and expect to continue to work without a hitch. Yet we cut back on them or use budget operators because... well... we take the risk and then bitch about but will still refuse to fund public transport or book the cheapest flight possible.

    That is why there isn't a good backup solution, you are not willing to pay for it.

    --

    MMO Quests are like orgasms:

    You may solo them, I prefer them in a group.

  25. you said it: scale by fantomas · · Score: 1

    You said it: the reason people flew ok in the 40s and 50s but can't now without computers is scale. There's just so much higher volume of air traffic, critical systems depend on computers to track too many things. Random 737s flying around a crowded air space unannounced could be a little dangerous, to say the least.

    If you reduced the number of flights to 40s levels probably you could do without computers. I guess this means your regional hub reducing to flying maybe one plane an hour with 20 passenger seats on each?

  26. Decentralize this. by drolli · · Score: 1

    Please, how about printing all the important information ad a 2D barcode on the tickets (cryptographically signed. of course), which can be read without a connection to the central system and having for the really important stuff = scheduling of planes a second independent system. I always wonder that the cost of these sw bugs could easily go into the tens of millions of $ , so it should be possible to take measures not to be completely dependent on a single point of failure.