Slashdot Mirror


Computer Glitch Friday Grounded US Airways Flights

mschaffer writes "A computer glitch Friday night snarled the travel plans of US Airways customers, as reports flooded in of flights grounded around the country." As someone stranded for several hours yesterday by this outage, "glitch" seems like quite a euphemism. With outgoing flights blocked, and new ones arriving full of passengers expecting to meet connections, the atmosphere got a little heated. Customers could see nice weather, and planes lined up outside, but "The System Is Down" trumps all. The E concourse at Charlotte (a US Airways hub) was packed full of customers ranging from livid (a handful) to merely angry (most) to calmly resigned — which means those of us with seats, snacks, and books or computers. It was disheartening to see how brittle is the infrastructure the airline employs; with the part of the system visible to airline employees down, customers thought they might get more information, or even rebooking, through the US Airways website. But that was down, too, and all the desk staff could do is shrug.

140 comments

  1. COBOL's fault, I'm sure! by Kensai7 · · Score: 4, Funny

    OK, let's count to three and blame COBOL! :p

    --
    "Sum Ergo Cogito"
    1. Re:COBOL's fault, I'm sure! by Anonymous Coward · · Score: 1

      Probably TPF running assembler, c and any number of Java connections

    2. Re:COBOL's fault, I'm sure! by rvw · · Score: 1

      Probably TPF running assembler, c and any number of Java connections

      TPF, what that? A Turbo Pascal Fuck?

    3. Re:COBOL's fault, I'm sure! by Anonymous Coward · · Score: 0

      TPF - no that's a proper operating system used by proper computers and proper programmers.

  2. umm... by datapharmer · · Score: 2, Interesting

    that sucks. No backup paper system in place? Can't they just read what the tickets say such as flight and seat number? They know where the flights are going as most are routine. It seems they should have been able to get *some* flights in the air.

    --
    Get a web developer
    1. Re:umm... by linest · · Score: 3, Interesting

      You can't fly unless you can prove your aircraft has had all required maintenance done. There are also rules about the number of hours per day crew members are allowed to be in the air. I suspect these records could be printed and used if it were a planned outage but this wasn't.

    2. Re:umm... by DesScorp · · Score: 5, Informative

      I work in airport IT, so I'll describe what I see airline crews doing during trouble. If the system at one gate or terminal is down, yes, then they'll send the plane on it's way. This is called "boarding manually". They simply hand collect tickets, hand count bags, etc, and send the flight off. After they've gathered all of the info thats been collected manually, they'll send it to their local office or front desk and process it at working terminals that have a connection to airline systems. It's a pain, but do-able. But if EVERYBODY is down, then the whole thing grinds to a halt. If no one has any access to all the schedule info, weight and baggage, manifests, etc.... then it's simply impossible to board manually on a massive scale.

      --
      Life is hard, and the world is cruel
    3. Re:umm... by DerekLyons · · Score: 2, Informative

      No backup paper system in place?

      One of the reasons they went to computers in the first place is because paper systems could no longer handle the workload... And that was back in the 60's when air traffic volumes were a fraction of what they are today. I.E. having to maintain a duplicate paper system would actually slow things down likely without actually providing sufficient backup.
       

      Can't they just read what the tickets say such as flight and seat number?

      That isn't much help with getting the luggage on the appropriate aircraft. Nor does it help to inform what flights (that you're expecting passengers from) are on time or nearly so and will or will not effect the flight in question. (Let alone routing the luggage involved.) Not to mention the number of passengers and the weight of the luggage - something the pilot needs to know to operate the aircraft safely.
       

      They know where the flights are going as most are routine.

      There's a lot more information flowing through the system than just "plane A goes to destination B" and "butt X goes into seat Y". With the system down they don't even know when/where plane 'A' is in order to get butt 'X' onto it.
       

      It seems they should have been able to get *some* flights in the air.

      No offense, but that's because you don't even remotely understand the problem. (And seemingly can't even be bothered to try by asking questions rather than making statements.)

    4. Re:umm... by Anonymous Coward · · Score: 1

      they were faxing the necessary paperwork. Was supposed to pick up a friend @ SFO coming from Charlotte. They stated the systems were down and they had to fax all the pilot's info in order for the plane to take off.

    5. Re:umm... by Anonymous Coward · · Score: 0

      > then it's simply impossible to board manually on a massive scale.

      So how did airlines work in, say, 1960?

    6. Re:umm... by FlyingGuy · · Score: 3, Interesting

      In 1960 most everyone was running Sabre

      Most everything was connected to hard lines that went back to the big main frame machines that ran it.

      If one terminal was down then it was most likely the terminal that had failed or possibly one of hundreds of hard lines back to the Main Frame

      Now days with everything being all cloudy good luck figuring out what might still be available. It could have been something as simple as the single bit of fiber serving that main concourse was damaged or a router someplace in the airport had failed or some cloud vendors routing had gone south or hell it might have been something as dramatic as what happened to Amazon a little bit ago.

      --
      Hey KID! Yeah you, get the fuck off my lawn!
    7. Re:umm... by Anonymous Coward · · Score: 0

      With a lot more people on hand trained to do tasks that are now automated. Them computers, they're stealing our JOBS!

    8. Re:umm... by Anonymous Coward · · Score: 0

      Fine, so replace the question with 1950, or 1940. There have been airlines for a long time and I'm sure they were able to board planes before they had computers.

      Granted not at the same scale that we have today, but they were not crippled by the lack of computers. Have people gotten dumber since then, and are no longer able to function without computers to do the thinking for them?

    9. Re:umm... by Chaos+Incarnate · · Score: 2

      They didn't have Homeland Security wiggling their fingers and watching the airlines and passengers dance like marionettes.

      --
      Benford's Corollary to Clarke's Law: "Any technology distinguishable from magic is insufficiently advanced."
    10. Re:umm... by h4rr4r · · Score: 1

      It's not the thinking that is the problem it is the record keeping.

      How do you check that the plane has had the required maintenance?
      How do you check that the pilot has not been flying too long today?

      Not like they have massive filing cabinets at all the airports for this stuff.

    11. Re:umm... by Grishnakh · · Score: 2

      The number of passengers per day in 1940 was next to nothing, and today it's staggering.

      Similarly, back in 1940 or 1950, "long distance" telephone calls (remember those?) were routed manually, by operators at switchboards. Can you even imagine trying to go back to that, with the sheer number of "long distance" telephone calls going on these days?

      When you do things of great complexity in huge volume, you need to have computers do them. Sure, it's possible to do them manually if you already have an army of trained personnel at hand, and equipment and processes in place for them to use and follow. But it's not like you can just go back to that in an instant. Just like there are no more manual operator switchboards at telephone companies to fall back to, there's no way to fall back to handling flights manually.

      The problem is that many things that are automated simply aren't designed with sufficient redundancy and reliability. Some high-profile outages will fix that though.

    12. Re:umm... by Trilkin · · Score: 1

      That's just it, though. The scale has increased DRAMATICALLY. Back in the early days of commercial flight, it was generally very expensive and there were very few people able to take advantage of it plus the population of people more than a little nervous of sitting in big flying tin cans. It was, thus, easy to keep records when you only have a handful of planes and passengers with practically no security.

      --
      Nobody cares what the CAPTCHA for your post was.
    13. Re:umm... by jd · · Score: 1

      I know Slashdotters don't always have the best manners, but this isn't Kuro5hin. We still have standards.

      Whilst I agree that paper backup is probably out of the question, most computers are quite capable of handling multiple ethernet lines and most routers are capable of supporting hot standby configurations. Even cold standby is a 30 second failover. The same goes for backend servers - it doesn't take much to add a checkpoint/failover system (cold standby) and it's quite possible to configure most servers to support hot standby.

      Asteroid takes out a data center? Well, then you've probably got bigger issues, but co-locating across the country is Standard Practice for most instustries.

      This simply isn't about the problem. It's about whether the solution has been implemented. Nothing more.

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
    14. Re:umm... by egburr · · Score: 1

      Overall, I agree with what you said. Only one part didn't make any sense, though: "With the system down they don't even know when/where plane 'A' is in order to get butt 'X' onto it."

      Couldn't they just look out the window? Maybe that big thing outside the window that a whole bunch of people recently walked off of just happens to be the plane everyone at the gate is waiting to board?

      Yeah, just loading it up and taking off would make a huge mess of the paperwork, but don't tell me they can't find the plane.

      --

      Edward Burr
      Having a smoking section in a restaurant is like having a peeing section in a swimming pool.
    15. Re:umm... by digitig · · Score: 4, Insightful

      This simply isn't about the problem. It's about whether the solution has been implemented. Nothing more.

      And that is down to whether it is cost-effective to implement the solution. You will never be able to get the probability of failure down to zero. and the cost skyrockets the closer you get to zero. How often do outages like this happen, and how much would it cost to prevent them at every airport worldwide? And to prevent every other conceivable scenario? Yes, it could have been prevented, and lots of other possible outages that didn't happen could have been prevented, but the cost of air tickets would be prohibitive.

      --
      Quidnam Latine loqui modo coepi?
    16. Re:umm... by Deadstick · · Score: 3, Interesting

      Rent an amusing sf film made in 1953 called "The Magnetic Monster" and watch an airport official tell Richard Carlson "We can't search ALL the flights. This is Los Angeles International; we have over a dozen departures a day!"

      rj

    17. Re:umm... by timothy · · Score: 1

      One of the things I saw yesterday was gate switches; the desk staff don't know until they get the (apparently very sporadic) updates about things like that, so people are often directed to distant gates rather than their originally scheduled ones. And when it happened yesterday, I saw two different flights (to different cities!) both being sent to the same gate, and I'm pretty sure they weren't both right. Glad I just had to wait, and it was my final leg.

      timothy

      --
      jrnl: http://tinyurl.com/c2l8yr / foes: http://tinyurl.com/ckjno5
    18. Re:umm... by Anonymous Coward · · Score: 0

      Don't fly much? I haven't had a "ticket" in ten years.

    19. Re:umm... by Anonymous Coward · · Score: 0

      How often do outages like this happen, and how much would it cost to prevent them at every airport worldwide?

      Power outages at data centers? Somewhere in the world, you probably have at least one a day.

      I understand when cities don't create specific plans for zombie invasions, but if you consider a power outage to be too rare an even to justify planning for,you're incompetent.

    20. Re:umm... by Anonymous Coward · · Score: 0

      And if the system could run on a simple hot standby system with a few ethernet connections and a few printers here and there, why in the hell would they have mainframes and full blown world wide networks? They don't just make the existing system complicated and that large for the hell of it.

      Anyway...
      Airlines have mainframes in a central location and usually a backup or some redundancy in another physical location. All of the airports have their own redundant connectivity back to the mothership and have nothing server based on site, not even the printing of tickets, manifests, maintenance procedures, pilot flight info or weather reports are generated locally, it is requested from the mainframe and it sent to a line printer attached to a specific computer that is addressed directly by the mainframe. That computer (at the gate, customer service counter, in the maintenance area, flight ops, baggage handling or where ever) is running some type of terminal emulation software and processes the requests to a serial port. The mainframes do run with probably 99.999 percent uptime, this was the .001% that it was down.

      The only servers you will find in an airport related to the process of getting a plane in the air will be local support systems like DHCP (if they are even using that), TFTP, the backend for the arrival/departure display screens and simple things like that. None of which would bring the airline at that airport to a halt if those failed.

    21. Re:umm... by X0563511 · · Score: 1

      Yay fax! A technology older than voice telephones.

      Why the fuck are we still using faxes?

      --
      For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
    22. Re:umm... by Anonymous Coward · · Score: 0

      I built a lot of applications such as ticketing, queue management, and baggage handling systems about 10 years ago and at that time there were 4 or 5 big airlines that actually shared some of the same systems. The systems were built by a company funded by the airlines. It would have been very unusual back then if only one airline was having a problem that did not also effect the others.

    23. Re:umm... by linest · · Score: 1

      Why the fuck are we still using faxes?

      Because the computers are down?

    24. Re:umm... by DerekLyons · · Score: 1

      I know Slashdotters don't always have the best manners, but this isn't Kuro5hin. We still have standards.

      They vary wildly between 'unreasonably high' and 'ludicrously amusing', but yes, Slashdot has standards. (And very occasionally, the have some relevance to the real world.)
       

      Asteroid takes out a data center? Well, then you've probably got bigger issues, but co-locating across the country is Standard Practice for most instustries.

      If 'most' industries had a system even half as complex - you'd have a point. But 'most' industries don't dynamically track dozens of airports, hundreds of aircraft, and tens of thousands of passengers (and their luggage) in real time. Worse yet, not only in this information dynamic, its also tightly interrelated. (That plane I'm meeting in Dallas after taking of from Sea-Tac this evening is the same plane that flew from LA to Chicago this morning, and then flew Chicago to Dallas.)
       
      This isn't just a simple matter of backing up data and providing redundant communications links... It's also a matter of keeping the hot spare updated and parallel to the active string at about the triple nine level. This is both very expensive and very hard. (And requires something much more than the standard PC and server mentality the average Slashdotter deals with.)
       

      This simply isn't about the problem. It's about whether the solution has been implemented. Nothing more.

      As with the poster to whom I replied, you don't even understand the problem. This results in facile and inappropriate 'solutions' that don't even begin to address the issues.

    25. Re:umm... by DerekLyons · · Score: 1

      Maybe that big thing outside the window that a whole bunch of people recently walked off of just happens to be the plane everyone at the gate is waiting to board?
       
      Yeah, just loading it up and taking off would make a huge mess of the paperwork, but don't tell me they can't find the plane.

      Assuming the plane is at the gate - what about an hour before when it's still a couple of hundred miles away? What happens if it's delayed, or never even takes off? How do they route the luggage since it doesn't have the tags that they can't print because the system is down?
       
      As I said before, not only do you not understand the problem, you can't be bothered to even try.

    26. Re:umm... by digitig · · Score: 1

      It wasn't just a simple power outage at a data centre, though, as far as I can see. And even if it was, it's still a cost-benefit analysis to decide whether it's worth doing anything about it. If you make all of your systems multiply redundant with no regard for the cost-benefit then you are not going to stay in business for long.

      --
      Quidnam Latine loqui modo coepi?
    27. Re:umm... by jd · · Score: 1

      Having worked in places with hundreds of remote offices, for that matter having worked at CERN on data collection for nuclear accelerators, I think I might, just might, have an idea of what it takes to keep large numbers of systems in sync over continental distances.

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
    28. Re:umm... by Anonymous Coward · · Score: 0

      So you think all systems are designed the same? That shows your lack of experience, not the high level of expereince you think you have.

      Remember back when airlines had dumb terminals with monochrome screens? They still have that exact system today, the only difference is the dumb terminals have been replaced with windows running terminal emulation software. They may be using a cool looking app on the screen and scanning barcodes etc when you board but the PC they are using or close by is translating that data to plain old text commands which the agent could type if they knew the syntax (which some old timers do but it would not help if the system they are talking to goes down anyway). If you want a redundant backend for that, you will need another mainframe in sync with the existing one which they already have. Airlines run these core systems at 4 and 5 9's already. Putting a completely different system in place to get more 9's is not cost effective or justified at all. There is NO way that a client server setup spread around across the world would be more reliable or redundant then the mainframe dumb terminal system they have now is. My guess, probably a software error that someone loaded on and caused problems.

    29. Re:umm... by Anonymous Coward · · Score: 0

      I hit submit too early..

      have an idea of what it takes to keep large numbers of systems in sync over continental distances

      That is great but unrelated. Every one of those computers at every ticket counter position, down in the maintenance area, the baggage loading area, the skycap stations, at the 1800 reservation center, and at the gates, along with the ticket printers, walk up kiosks, bag tag printers etc is attached directly to the mainframe backend as an addressable remote console so there is nothing to synchronize or get out of sync ever. They are all directly interfacing with a single master system (or multiple systems load balanced and geographically separated). Nothing is processed at the local airports.

    30. Re:umm... by jd · · Score: 1

      It's a huge transactional database, yes. What's your point?

      When building a fault-tolerant system you've two choices:

      a) Build the system such that you have N databases (not 1) with transactions sent to any of them replicated across the others. By doing transactions in bulk over some period (say every 5 mins), everything within that 5 min window will always be in the correct order on all instances of the system. By having the kiosks retain the past 5 mins of transactions and the electronic address of each server, if one goes down it can round-robin to the next and dump the buffer into it.

      b) Use NACK-oriented reliable multicast and have each transaction automatically forwarded to all N servers at the same time.

      You use leased lines and MPLS to create the virtual circuits needed for this. One fully reliable, trans-continental system. Because each kiosk "sees" one server (even though it may change at any given time) and that server is always current, it "sees" a single master system even though no master system exists.

      Look, this is stuff I know you youngsters have a hard time understanding, but my generation had already solved these problems before half of the current Slashdot population was born. Now gerroff my lawn!

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
    31. Re:umm... by FlyingGuy · · Score: 1

      Not to mention hoping the pilot gets the weight & balance correct, not to mention fuel consumption ( which none of them have had to calculate in years ). Most times the plane datalink has the correct information but I have seen a commercial jet parked at the gate because the feed from the computer was down.

      Making assumptions based on the standard FAA passenger w/standard baggage is tricky business not to mention dangerous since that has killed more then a few passengers and crew especially stops at regional airports where some get off some get on and off we go! Out of range CG kills people.

      --
      Hey KID! Yeah you, get the fuck off my lawn!
    32. Re:umm... by mcmonkey · · Score: 1

      Fine, so replace the question with 1950, or 1940. There have been airlines for a long time and I'm sure they were able to board planes before they had computers.

      Remember the recent story on the Harvard entrance exam from thousands of years ago? I do, and I don't recall too many people bragging about how well they would have done on the Latin and Greek sections. Does that mean thousands of years ago people were geniuses, and the modern /. poster is an idiot?

      No. It just means people don't routinely study Latin and Greek anymore.

      Likewise, whatever system the airlines were using pre-networked computers may very well work today to keep things moving. Except no one working the terminal remembers that system.

      So are you proposing the hundreds of people working the terminal reinvent that pre-computer system? Each one on their own? Or shall they all abandon the stranded passengers, and come together to devise a replacement for the down systems?

      Remember, those folks who ran things without computers, they still had training and a system provided for them. They weren't just making things up on the fly.

  3. Shit happens by Osgeld · · Score: 1

    sometimes its a pain

    1. Re:Shit happens by Anonymous Coward · · Score: 1

      I'd just like to ask timothy to revisit this post after he's had some months or years. Being stranded for several hours is pretty frickin' minor as far as bad things that can happen when flying or in life in general; calling it a 'glitch' isn't much of a euphemism, although I might call it a 'major glitch'. Bad things seem much worse when they happen to you, but when you're an adult you're supposed to be able to get some perspective on it.

    2. Re:Shit happens by timothy · · Score: 1

      Yep -- I was fine.

      Like I said: snack, seat, plenty to read ... my Slashdot work for the day was done, too, so no one else was being much inconvenienced by my travel delays in particular. (It's the other people, with connections etc, who had bigger hassles.)

      This is so far from the worst thing that could happen even among modern travel disruptions that I hope you take my account with the same viewpoint I had: it's a 21st century problem / first-world problem, and that's the best kind of problem to have. Once I was stuck in San Antonio for 3 days because of ice storms ... which was actually pretty fun, since I wasn't missing a friend's last moments, an organ transplant, etc.

      The worst aspect for those people who didn't have cliff bars, gum, etc. is the uncertainty -- don't know if you have time to go snag an airport-priced sandwich, because you might miss an important announcement.

      And my favorite on this sort of problem remains this: http://www.youtube.com/watch?v=8r1CZTLk-Gk

      --
      jrnl: http://tinyurl.com/c2l8yr / foes: http://tinyurl.com/ckjno5
    3. Re:Shit happens by russotto · · Score: 0

      I'd just like to ask timothy to revisit this post after he's had some months or years. Being stranded for several hours is pretty frickin' minor as far as bad things that can happen when flying or in life in general; calling it a 'glitch' isn't much of a euphemism, although I might call it a 'major glitch'. Bad things seem much worse when they happen to you, but when you're an adult you're supposed to be able to get some perspective on it.

      No, when you're an adult you're supposed to be able to take whatever crap everyone gives out and just say "it's fine, treat me like the rug I am". This is called "maturity".

  4. Us air by jbrodkin · · Score: 1

    Ive had more problems with us air and united than any other airline. Theyre incompetent

    1. Re:Us air by oneiros27 · · Score: 1

      And that would match with J.D. Power's customer satsfaction reports for 2011:

              http://www.jdpower.com/travel/ratings/airline-ratings/traditional/

      Although, when you compare the "Low Cost" airlines, there's a few others that had similar bad ratings:

              http://www.jdpower.com/travel/ratings/airline-ratings/low-cost/

      --
      Build it, and they will come^Hplain.
    2. Re:Us air by Anonymous Coward · · Score: 0

      It's like eating out, I don't do it if I can't afford to tip. I pay extra to avoid United.

    3. Re:Us air by Anonymous Coward · · Score: 0

      Strange, I fly United out of Chicago whenever I travel cause it is my hub and have had the least number of issues (not related to weather) of them all. Let me guess you take United from some place that is not the hub and have to connect at a hub. You do not give yourself enough time to connect between flights. The system does not work that well and you need to increase your time buffer.

      AA is horrible for jacking flights around. Get there for the 7:00pm, gets canceled pushed to 8. Gets cancelled pushed to 9. End up flying the 9pm cause it is the last flight. Now i have spent 4 hours at airport.

      The worst problem I ever had was a full day trying to get to Cleveland on Continental (I know, they merged with UAL, not happy). Plane came in over hours on crew so they bused me to Milwaukee where that flight came in broken. Then got on the 4pm flight to Cleveland which left late. Luggage was still tagged for the 12:30 broken flight. Got to Cleveland at 7pm and luggage arrived at 11pm. Did I mention that I had a 9am flight back in Chicago to begin with.

      Like all big companies, airlines are a joke of operations and systems.

    4. Re:Us air by nwf · · Score: 1

      And it would match Consumer Reports who rated usair the worst of any domestic carrier. I'd link, but they keep almost everything behind a paywall. Usair used to be really good, until they merges with America West. United is second to last and that's who I'm waiting for now at the airport. Only 75 min late currently.

      Since united and continental are merging, I'll bet the combined airline will be far worse than either alone.

      --
      I don't know, but it works for me.
    5. Re:Us air by ShakaUVM · · Score: 1

      Yeah, that's a good point. If I lived in Atlanta, I'd probably fly Delta more often.

      Never had complaints with United. I fly a couple times a month. Well - we had a hydraulic issue once, they deplaned us and gave us free food and drink. Pretty reasonable, really.

      United scores points in my book for EcobomyPlus service. Not many others offer the legroom of first class for free(ish).

      My hatred for USAir is incomparable, though.

    6. Re:Us air by Drathos · · Score: 1

      I've never had any major problems with USAirways (a few delays here and there, but only one more than 30 mins). I actually flew through CLT yesterday, but I guess I got out before the problems started.

      Delta, on the other hand, is a constant problem. Left stranded in ATL twice, many delays, and terrible customer serice. Maybe if they didn't overload ATL, it wouldn't be so bad (that doesn't help the customer service issues, but it might with the others). I avoid them whenever possible.

      --
      End of line..
  5. Backups? by fysdt · · Score: 1

    Why isn't there a backup available in case a glitch occurs?

    1. Re:Backups? by RdeCourtney · · Score: 2

      Because they've calculated the customer apathy and money lost is less than implementing backup procedures. Remember it's all about $.

      --
      Insert signature here...
    2. Re:Backups? by Anonymous Coward · · Score: 0

      and this "backup" you mention. What would it do, exactly?

    3. Re:Backups? by CohibaVancouver · · Score: 1

      Why isn't there a backup available in case a glitch occurs?

      It's called "risk management." Let's say a backup system would cost $150M over 20 years, and the current system is calculated to fail every seven years, at a cost per failure of $30M (cost in terms of lost business / OT / brand damage etc.). Running without the backup system you're many tens-of-millions of dollars ahead in the game. These figures are just made up, but these sorts of calculations go on all the time in many different industries.

    4. Re:Backups? by linest · · Score: 2

      Because they've calculated the customer apathy and money lost is less than implementing backup procedures. Remember it's all about $.

      I'm not crazy about the way that's phrased, but you are essentially correct. Establishing backup data centers, populating them with hardware, purchasing additional software licenses, establishing, testing and maintaining fail over procedures is nontrivial. When you consider the overall health of the airline industry, it's not surprising that the extra tens of millions of dollars were not spent.

      It'd be interesting to know how many millions of dollars this will end up costing US Airways. I'll bet accepting the problem saves money over solving it. If you had a car worth $2000, you wouldn't spend $10000 to insure it. That's a rational decision.

    5. Re:Backups? by jd · · Score: 1

      That depends on how you define cost. The instantaneous cost is one part, but only a part. Even so, that cost isn't just frustrated customers. It's parking costs for the aircraft, staff wages, any technician overtime needed, costs due to food spoilage, loss of in-flight sales, etc. Delayed costs also matter. There's any loss of future custom to consider, since that is also a cost to the company. There's any increase in insurance costs for them as a result of any successful claims. It may well impact the airline's ability to purchase space at an airport or purchase a specific route. All these things are costs.

      The problem is that many aren't quantifiable - too many unknowns - so an airline is incapable of knowing if a backup system is cheaper or not.

      Oh, as for additional software licenses, many enterprise-level software vendors support floating licenses. So if you're doing cold standby (a whopping 30 seconds of outage), no additional licenses are needed.

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
    6. Re:Backups? by jd · · Score: 1

      Fail over the TCP/IP connections (hot standby) or recover from a checkpoint (cold standby), re-sync with events and then continue. Why do you ask?

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
    7. Re:Backups? by Sponge+Bath · · Score: 1

      There was a backup plan in the suit pocket of the CIO, but it was casual Friday, so everyone was, you know, wearing a Hawaiian shirt.

    8. Re:Backups? by Anonymous Coward · · Score: 0

      What if it wasn't a network/traffic issue? No point in failing over if the root cause is further up the pipe?
      Recover from a checkpoint - very simple to say - but what if something has happened that has longer reaching issues than a simple recover/re-sync? What if the recover takes a long time? What if the backups are incomplete? What if? What if?

      Then what do we do?

      Your suggestions are very high level. Very easy to write down, but the devil is in the detail. As you don't know the detail, quotes like "fail over the TCP/IP connections" are ... well, worthless.

    9. Re:Backups? by digitig · · Score: 1

      The problem is that many aren't quantifiable - too many unknowns - so an airline is incapable of knowing if a backup system is cheaper or not.

      But the airline still has to make the call. Pay for a backup for system x or take the hit if it fails. And they can only sensibly make that call if they can make an estimate of how much the hit will cost and how often it's going to happen. And don't forget: multiply redundant systems can fail, too. I've seen a power system based on main and standby UPSs fed by different power supply companies, backed up by main and standby generator sets, each of which comprising main and two standby generators, go down completely at an air traffic control centre (due to one maintenance issue and one design error). Just how much do you want the airlines and airport to spend to make sure something like this doesn't happen? How much are you willing to pay for your ticket?

      --
      Quidnam Latine loqui modo coepi?
    10. Re:Backups? by Anonymous Coward · · Score: 0

      You give them too much credit. Re-do that with the 7-year failures at $100M and you'll still not get the backup system. "You tellin me that I gotta spend $150M on a backup system? Does that mean our current system is inadequate?" "No the system is fine, but we'd like $150M for a backup anyway." "Uh, no." It doesn't matter how many studies you show. Upper management will always think that it's lies from the IT department to expand their budget or lies from contractors to pad their pocketbooks. And it's quite amusing when you are held responsible with the email chain begging for the backup system and being denied when the failure does come and upper management blames you. Why amusing? Because the upper management will look at the same email chain and point out where you said there were no problems with the current system, and it failed, so you were obviously wrong.

      It takes not only accurate risk numbers (and those are nearly impossible to get for anything more complex than a single HDD), but upper management willing to listen to the risks and costs, rather than just the costs.

    11. Re:Backups? by linest · · Score: 1

      The problem is that many aren't quantifiable - too many unknowns - so an airline is incapable of knowing if a backup system is cheaper or not.

      I like that observation a lot. It could be carried further. You're talking about actual risks. Real decisions are made based upon perceived risks that sometimes consist of little more than assumptions. Especially once you get out of the IT realm and need something paid for. Since these are IT risks and they need to be communicated to non-IT people, there is a challenge there. It's not easy.

      Your comment about software licenses being free for disaster recovery, on the other hand, seemed a bit too off hand. I believe the issues are the databases for crew scheduling and aircraft maintenance. That's the stuff that'll keep you on the ground. To my knowledge, there is no dominant application for aircraft maintenance packages. If that data is stored in an Oracle database, you're going to pay big bucks for DR licenses. On the other hand, if we assume that US airways is using a Jeppesen package for crew scheduling (I'd bet a small amount of money on this), then it relies on a 10 year old version of Informix. Disaster recovery for the database server software would (in my experience) be free. IBM's OK that way.

  6. Very interesting by jhoegl · · Score: 1

    I dont know what the "outage" was, but it seems redundancy is an afterthought with US Airways.
    So, I can assume the following
    There is no redundancy(zing)
    There is no Recovery plan
    There is no DR plan
    There is no SoP on releases
    There is no SoP on testing
    There is no SoP on handling outages
    Bravo! Now, did those new TSA rules go into effect yet? Can US Airways be fined into oblivion because of this?

    Also, what are the extra fees US Airways charged its passengers for having to handle their complaints and angry faces? $100 per complaint?

    1. Re:Very interesting by Anonymous Coward · · Score: 0

      All this backup shit eats into profits ^H^H^H^H^ are not revenue-generating measures. The shareholders must be thought of as well, which as economists remind us constantly, are us, the 401k holders, and definitely not 50% owned by the richest 1%. That would be unprintably deceptive.

      Management has a well-proven disaster recovery plan in place; lobby for a bail-out.

    2. Re:Very interesting by brusk · · Score: 3, Insightful

      Would you buy tickets that cost $25 than anywhere else more because the airline advertised redundancy in its IT systems? People choose flights based on price (secondarily, on frequent flier plans, etc.), but how is a consumer supposed to choose based on factors like this, except in the most general terms (on-time percentage)? The airline management knows this, and would be silly to invest too much in things that will raise costs without enabling them to increase revenues.

      --
      .sig withheld by request
    3. Re:Very interesting by jd · · Score: 1

      Airlines with outstanding reputation for timliness and customer service probably could charge $25-$50 more per ticket and have the customers grateful for it. The problem is that it takes decades to build that kind of rep but mere seconds to destroy it. Much easier to pretend to cater for the unwashed masses because that means guaranteed profit now rather than a higher but riskier long-tern profit due to good, competent service.

      It's the way Microsoft, T-Mobile, Comcast and talk-show hosts have bilked people for years.

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
    4. Re:Very interesting by Idbar · · Score: 1

      I like to check flightstats and check if the flights I'm picking have bad statistics about being late most of the time. In that sense I try to avoid stopping in Denver and sometimes Las Vegas.

      I know stuff happens, but if I can minimize the amount of things that can go wrong, I'm willing to pay a bit more.

    5. Re:Very interesting by ShakaUVM · · Score: 1

      People that would pay $25 for a better flying service are NOT flying US Airways anyway. It has the worst customer service of all the legacy carriers.

    6. Re:Very interesting by digitig · · Score: 1

      Airlines with outstanding reputation for timliness and customer service probably could charge $25-$50 more per ticket and have the customers grateful for it.

      Yes, and other customers will choose the airline that charges $25-$50 less per ticket to save the money, then gripe when they get on the wrong side of the inferior reliability.

      --
      Quidnam Latine loqui modo coepi?
    7. Re:Very interesting by sgtrock · · Score: 1

      It's the way Microsoft, T-Mobile, Comcast and talk-show hosts have bilked people for years.

      You're going to single out T-Mobile??? The one carrier in the U.S. that actually had an HTC Android phone as soon as it was released? The one carrier in the U.S. that actually seems to give a damn about not loading up their phones with all the garbage the others do? The carrier in the U.S. who doesn't make life hell if you want to use a phone that you bought some place else?

      Granted, their coverage for data outside major metropolitan areas can be spotty, especially west of the Mississippi. Their coverage maps make no secret of that fact. It's a LONG leap not being satisfied with their coverage to putting them in the same category as Microsoft and Comcast. If any of the U.S. carriers belong there, it's the big three; AT&T, Sprint, and Verizon!

    8. Re:Very interesting by drsmithy · · Score: 1

      Airlines with outstanding reputation for timliness and customer service probably could charge $25-$50 more per ticket and have the customers grateful for it.

      They couldn't, because the vast majority of their customers don't fly frequently enough to care, and/or purchase solely based on price.

      People who do fly frequently enough to care, are generally protected and/or bought off by their frequent flyer status (frequent flyer programs in the US are _very_ generous).

    9. Re:Very interesting by 19061969 · · Score: 1

      I pay a lot more than $25 extra to fly with airlines that don't keep pissing me and my family around with seat re-allocations, and I've often avoided poorer quality airlines offering cheap prices).

      Price is a factor but it's not the only one. Availability is also a factor as are length of stop-overs, chance of getting bumped up, total flight length, where I have to transit and so on.

      Proviso: I rarely fly in the US and my airlines of choice are Singapore, Emirates and Air NZ (the latter of which are often way too expensive so I prefer to go SK or EK which are so far ahead of US carriers it's not real. It's strange considering the US usually has excellent levels of customer service elsewhere why it's so poor for air travel. I guess I'm lucky in that I can often avoid US airlines if traveling to the US.

      --
      bang goes my karma... again...
  7. Computer Glitch Friday? by QuasiSteve · · Score: 1

    I know IT fully embraced Patch Tuesday leaving us with up to a month's worth of accumulated crud, but now they've gone too far!

    1. Re:Computer Glitch Friday? by Beelzebud · · Score: 1

      I read it exactly the same way. I guess using "A" and "on" was too much extra typing. :D

    2. Re:Computer Glitch Friday? by jd · · Score: 1

      Maybe the computer glitch is an artificial intelligence called Friday. It then called up the airports and ordered the planes not to take off.

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
  8. Wait, what? by Anonymous Coward · · Score: 2

    FTFA: "The Tempe, Ariz-based carrier cited a power outage near one of the airline's data centers in Phoenix as a possible cause."

    A POWER OUTAGE?! So, no UPSes, no generators, and no multiple utilities at a main data center for a major company? Come on now...

    1. Re:Wait, what? by jd · · Score: 0

      Seems a fashionable type of problem to have.

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
  9. Snarled? by Anonymous Coward · · Score: 0

    Snarled? What is 'snarled'?

    1. Re:Snarled? by dtmos · · Score: 1
  10. What?! by Anonymous Coward · · Score: 1, Insightful

    I love it when people melt down and scream and yell. Fucking christ. Shit happens. Shut your mouth, go over to the airport Starbucks and buy yourself an overpriced airport coffee and calm down. You're not helping. You're not the center of the universe, you twit; in fact, they're probably not even going to even up the scales.

    (I'm one of those in the calm catagory when it comes to Emergencies, or (more likely) "emergencies.")

    1. Re:What?! by egburr · · Score: 1

      A cup full of caffeine and sugar is supposed to help you calm down?

      I used to carry a couple extra books in my carry-on bag, as I have seldom had a flight even come close to being on time (except for connecting flights which are almost always on time no matter how late my initial flight is).More recently, I just make sure my phone charger is with me, so I can keep my battery topped off as I read ebooks or play games on my phone instead. My only real complaints are that I have yet to find an airport with even slightly comfortable seating and that there is never any place at all to get away from all the noise of the multiple TVs tuned to multiple stations all at full volume and the incessant security announcements just in case there is someone in the airport who hasn't flown in the past 20 years.

      --

      Edward Burr
      Having a smoking section in a restaurant is like having a peeing section in a swimming pool.
    2. Re:What?! by KingAlanI · · Score: 1

      Yeah, I also thought that Starbucks wasn't the best example.
      Last I I flew, it was for once it was good that I tend to overpack, as that in part included extra reading material that I hadn't gotten to the rest of the trip.

      --
      I listen to both RIAA and non-RIAA stuff if I like the music, tangential business/politics nonwithstanding.
    3. Re:What?! by Abstrackt · · Score: 1

      My only real complaints are that I have yet to find an airport with even slightly comfortable seating and that there is never any place at all to get away from all the noise of the multiple TVs tuned to multiple stations all at full volume and the incessant security announcements just in case there is someone in the airport who hasn't flown in the past 20 years.

      HHGTTG actually got this one right: bring a towel. You can use it as a cushion on an uncomfortable seat or even as a pillow if you want to nap on the floor. Just make sure it's a thinner towel, ideally a travel one, so it doesn't take up too much space in your bag/suitcase. As for the noise, IEMs are a godsend. Once my music or ocean sounds are on I don't hear anyone or anything else, perfect for those six-plus hour stopovers.

      --
      They say a little knowledge is a dangerous thing, but it's not one half so bad as a lot of ignorance. - Terry Pratchett
  11. United was also FUBAR yesterday... by neurocutie · · Score: 1

    I specifically chose United over USAir for travel yesterday as I've had the most trouble with them. However United was also in poor shape yesterday. It was termed 'operational delays', with two hour delays across the board. Calls into United faced 25-30min wait times. And many overbooked flights.

    Seems the whole industry is going down the tubes... and decreased competition from these mega-mergers are not helping.

    1. Re:United was also FUBAR yesterday... by Anonymous Coward · · Score: 0

      When one airline has a major issue, it sends shock-waves throughout the entire industry. Those passengers that were delayed or canceled needed new adjusted flight times. Often, that changes the schedules of all those individuals affected. If I had to guess, there was a major spill-over from USAir passengers to United and the entire air traffic control process all gummed up.

  12. rise of the machines by Anonymous Coward · · Score: 0

    its over folks

  13. System failures only affect large airports by Anonymous Coward · · Score: 0

    If you have a tiny airport near you, its better to use that (if practical)

    recent experience: at Mangalore airport (IXE), The systems were "down"

    Steps followed by management:
    Handwritten boarding passes
    For people with connections they actually called up the airport at which they were taking a new flight,(for each passenger) and had a SPOC set up for each airline at those airports.
    Now, this airport handles less than 15 flights per day, but 90% of those are college students, just starting their vacations, so you can imagine the mess that would have resulted..(but it didnt due to the improvisation)

  14. Right by Anonymous Coward · · Score: 0

    All you who choose to fly others will fly who is ever cheaper that minute. It all goes full circle. Stuff happens, deal with it.

  15. Could you tell the difference? by Cutriss · · Score: 4, Interesting

    As I am now located in proximity to an airport with a US Airways "service focus" and have had the "pleasure" of flying with them several times, I have to ask - how would you be able to tell the difference? Every time I've been in a US Airways terminal, there's always a significant number of non-weather-related delays and cancellations (compared to the other airlines' monitors). My wife and I have independently had three separate incidents this year where we were 4th and inches from having to stay overnight at an airport due to cancellations/late planes/overbooked crew/etc. In two of those cases, I had flights where we took off at the 2'55" mark, just shy of the three hour requirement to return to gate and let everyone off. The cynic in me suspects that US Airways is actually using that three hour window to plan its flights.

    It's an abhorrent mess, and when I see the US Airways CEO defending against his last place customer service ranking, I have to wonder just how much denial one management team can stand.

    --
    "Mod, mod, mod...and another troll bites the dust."
    1. Re:Could you tell the difference? by Anonymous Coward · · Score: 0

      As a very frequent flyer, I no longer fly on US Scareways. I've been subjected to being trapped for 4 and a half hours at an airport on one of their flights and not being allowed to disembark because of "liability concerns." I have been on US Air flights that have flown into severe weather when they could have easily flown around it. There was one flight when there was an aborted take off from Philly a couple of years ago. After a 3 hour sit on the plane waiting for rain to pass. After the aborted take off they then canceled the flight because of weather. Never again, they are fucking retards of the highest order.

        They are last place because they suck on so many levels and I don't care if I have to spend 5 times the cost on other airlines to fly, I won't fly with these idiots any further.

      Last year I donated 40,000 US Scaremiles to charity because I wouldn't subject anybody I know to a flight on their airline.

    2. Re:Could you tell the difference? by jittles · · Score: 1

      See I have the exact opposite experience with US Air. Free upgrades to first class, and have not experienced any more delays than any other airlines. My worst experiences have been with Delta and AA.

    3. Re:Could you tell the difference? by Anonymous Coward · · Score: 0

      Ahh, Charlotte-Douglas (either that or Cincinatti, but I'll guess Charlotte, since seem to have no other options).... Most of my friends and coworkers make the drive to Greenville - lower rates, more choice, and (bonus) you usually layover in Charlotte. So, have a friend take you to Greenville, and then just disembark in Charlotte on the way home.

      I've been here 12 years now, and I've hoped and hoped for improvement, but alas, none has come.

  16. Brittleness is a pervasive problem in air travel by Anonymous Coward · · Score: 0

    It is the entire industry, not just their (reliance on) their computer systems, which is brittle. They set schedules to minimize costs, which, in part, means scheduling every aircraft as intensively as they can get away with. This leaves no leeway when something goes wrong. Your inbound aircraft is late, your outbound flight will be later. Weather or maintenance issues mean your aircraft can't get to your airport? There's no "spare" to bring online to cover the gap. Plus, because they're reducing schedules in order to make individual flights fuller, there are fewer vacant seats to absorb people affected by other mishaps.

  17. Whatever did we do... by Anonymous Coward · · Score: 2, Insightful

    ...before computers came along and made our lives so much -easier-?

    1. Re:Whatever did we do... by Anonymous Coward · · Score: 0

      Well, we didn't fly then.
      The age of mass public air transport was made possible by the age of computers.

    2. Re:Whatever did we do... by jd · · Score: 1

      I dunno. The R100 was a flying hotel and a fleet of those would have been quite capable of carrying the same number of passengers modern airlines could.

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
    3. Re:Whatever did we do... by 246o1 · · Score: 3, Insightful

      ...before computers came along and made our lives so much -easier-?

      Not fly anywhere, because it was too expensive.

      --
      Although the moon is smaller than the earth, it is farther away.
    4. Re:Whatever did we do... by blair1q · · Score: 1

      they weren't supposed to make your life easier.

      they were supposed to make your ticket price leave more profit in the airline's pocket.

      all those people who couldn't get anywhere? just how many got their money back?

      the system is working exactly as designed, even if the passengers don't get to feel all special about it.

    5. Re:Whatever did we do... by Anonymous Coward · · Score: 0

      So prior to the advent of computerized airline ticketing, no one flew?

      There's many years of history that beg to differ with ya.

    6. Re:Whatever did we do... by DerekLyons · · Score: 2

      Back in the 60's when airlines started computerizing, air traffic volume was a fraction of what it is today. You'd have to be nearly fifty to have even been alive at a time when computers weren't starting to make our lives easier in a variety of ways. ("Computers" !=" PC's".)
       
      I remember trying to make airline reservations back before the web. You couldn't pay me enough to go back to those days. (Unless you could also give me my twenty year old body as well.)

    7. Re:Whatever did we do... by Anonymous Coward · · Score: 0

      So no one flew before the advent of computerized ticketing? ::facepalm::

  18. No computer glitches by Anonymous Coward · · Score: 0

    Why isn't there a backup available in case a glitch occurs?

    Why isn't there a backup available in case a programmer fucks something up?

    FTFY.

    There are no "computer glitches" only human mistakes: programming error, design flaws, or data entry mistakes - none of which are the computer's fault.

    I just get peeved when customer service reps blame the computer thereby costing me money and time because someone on their end fucked up. If I arrived late and missed my flight because of a "watch glitch" you can bet your ass that they'll be the first to say, "That's not our problem! Now cough up the change fee!"

    1. Re:No computer glitches by jd · · Score: 1

      Good software is fault-tolerant. Fault-tolerant software DOES have a backup strategy if the programmer screws up. In the modern world, standards for software have fallen, not risen. If they had risen, virtually all software would be fault-tolerant and this kind of problem would not exist.

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
    2. Re:No computer glitches by fast+turtle · · Score: 1

      who said it was software. What about some idiot with a backhoe that cuts a cable?

      What I don't understand is why in hell a major hub airport doesn't have something like a system that's setup to cache this data? To my mind, this would actually help in the case some idiot with a backhoe does cut a com line or they get hit with some disaster like a tornado, huricane or god forbid an earthquake taking out large swathes of infrastructure.

      --
      Mod me up/Mod me down: I wont frown as I've no crown
  19. Referenced article later mentions a POWER outage.. by Anonymous Coward · · Score: 0

    leaving me to wonder whether it was a "computer glitch" or a simple power outage. /. posters are still right, backups should be in place regardless - and yes, there is a cost associated with that which some companies choose to forego. Does anyone else get the feeling that the news is worse on weekends, when junior people with less ability to spell, get the facts, or even understand what they are reporting, are writing the copy?

  20. I think Louis CK sums it up well by Time_Ngler · · Score: 1
  21. Last, and Dead Last by xkr · · Score: 3, Insightful

    I fly US Airways regularly. Last flight out was late taking off for no apparent reason. Our luggage did not make the connection in their own Hub. Neither did anybody else's. It took over an hour for the luggage clerk to process the long line. I counted over 500 keystrokes required per person. Staff didn't care at either airport. They would not put out luggage on the next plane in (another airlines, and they would have to pay a fee to that airline) so it was over a day to get out luggage. Two days, or three, unless we came back to the airport to pick it up. On the way home to SFO, it took over an hour for them to get out luggage onto the carousel. They had the nerve, over the PA system, to blame the passengers for having, "too much luggage," for the delay.

    Consumer Reports rated US Airways at the bottom of customer satisfaction.

    Planes fly. Southwest regularly makes last second changes, including flag stops (unscheduled) and re-using planes for "second runs."

    There was LOTS that US Airways could have done. First, they could have flown the planes if they wanted too. They planes had already been scheduled, so there were no questions of maintenance or fuel, or flight plans. Second, they could reimburse passengers for the delays. Third, they could have rescheduled some passenger.

    Then, of course, as said, there is simply no excuse for the IT to be down for that long, if at all. They had no (working) backup systems, either computers, paper, or people. That is the very definition of incompetent.

    I work in IT. As a guy said in my last meeting, “Anybody who designs in RAID 5 should be shot.” Duh.

    The fact is that the airlines management is incompetent. This is not an opinion. Simply too many facts. The board should completely clean house. When the questions comes up in the next board meeting of, “What to do?” the answer is, “Duh.”

    --
    I will create a sig when innovation restarts in the U.S.
    1. Re:Last, and Dead Last by T-Bucket · · Score: 1

      You are obviously clueless as to how an airline is run.

      They could not have just "flown the planes if they wanted to". Despite scheduling the flights, EVERYTHING is handled by computers now. Those flight plans you mentioned? Yep, filed by a dispatcher USING A COMPUTER. The performance calculations that determine how much fuel that flight plan will require? Yep, computer.

      Add to this the fact that the computers also control gate assignment, weight and balance, baggage routing, etc etc. There is NO WAY a modern airline can run their entire operation without computers. PERIOD. It's just not possible.

      (And yes, I do fly for an airline.)

    2. Re:Last, and Dead Last by Xacid · · Score: 1

      “Anybody who designs in RAID 5 should be shot."

      This runs counter to everything I've been taught - so perhaps you have some real world advice to support this? Merely curious.

    3. Re:Last, and Dead Last by Idbar · · Score: 1

      Even worse, they have managed now to blame anything and everything on weather so they don't have to reimburse people. Such that when they should have a properly deicer working (but they don't), they'd blame it on the cold weather.

    4. Re:Last, and Dead Last by IQgryn · · Score: 1

      RAID 5 is good at losing a second disk soon after the first, even with a hot spare (which means you lose the array). I think the GP meant something like "RAID 6 should be the bare minimum", or perhaps "any level of RAID is not as good as having an independent backup system".

      But I'd like to know what they actually meant, too.

    5. Re:Last, and Dead Last by Anonymous Coward · · Score: 0

      You were probably taught wrong then.. RAID5 is everything that's wrong with RAID, with only a little bit of the good. Need speed? Not for writing hopefully. Need redundancy? Oops, more than 1 disk failed?

    6. Re:Last, and Dead Last by Anonymous Coward · · Score: 0

      Reimburse This!...you buy a cheap (cheaper now than 30 years ago!) $300 ticket and if there is a hickup you want repramations to the tune of $500 and wonder why the airlines are charging for everything!

    7. Re:Last, and Dead Last by Anonymous Coward · · Score: 0

      RAID is not a backup. It's a form of redundancy, but that's it. RAID is also not a replacement for a proper fail-over cluster.

    8. Re:Last, and Dead Last by Anonymous Coward · · Score: 0

      Raid 0 for the win!

    9. Re:Last, and Dead Last by Anonymous Coward · · Score: 0

      Join BAARF -- http://www.miracleas.com/BAARF/BAARF2.html and you will understand.

    10. Re:Last, and Dead Last by Anonymous Coward · · Score: 0

      RAID 5 is good at losing a second disk soon after the first, even with a hot spare (which means you lose the array). I think the GP meant something like "RAID 6 should be the bare minimum", or perhaps "any level of RAID is not as good as having an independent backup system".

      But I'd like to know what they actually meant, too.

      “Anybody who designs in RAID 5 should be shot." Person making the comment is probably a software developer that thinks infrastructure people are "failed developers". His statement after this was probably something about "the cloud"

    11. Re:Last, and Dead Last by swalve · · Score: 1

      Yeah, OMG, RAID5 doesn't do things it wasn't designed for! Use proper hardware (Proliant/Dell) and it works just fine for what it is. A cheap way to drastically improve uptime. RAID isn't about anything more than improving uptime. And I've NEVER seen double disk failures in systems using said hardware and that had someone walk past the machine once a week or so looking for orange lights. ONCE I saw an array freak out because during a rebuild, a bad spot was discovered on another hard drive. But it was a huge array and really should have been raid6, and this would have been discovered if they were doing regular raid integrity checks.

    12. Re:Last, and Dead Last by Anonymous Coward · · Score: 0

      I interpreted it as "We don't need (failover | backups | redundancy), we have RAID 5, so it'll never fail!"

    13. Re:Last, and Dead Last by DerekLyons · · Score: 1

      There was LOTS that US Airways could have done. First, they could have flown the planes if they wanted too. They planes had already been scheduled, so there were no questions of maintenance or fuel, or flight plans.

      If it were a static problem, you'd have a point. But it's a dynamic problem involving dozens of airports, hundreds of aircraft, and tens of thousands of passengers and their luggage. Not to mention that flight plans are filed immediately before departure... so, no system, no flight plan. Not to mention that fuel calculations are performed shortly before departure, again - no system, no fuel calculations, unsafe as hell to fly. Not to mention that maintenance occurs constantly, so... well, hopefully by now you get the point.
       

      Then, of course, as said, there is simply no excuse for the IT to be down for that long, if at all. They had no (working) backup systems, either computers, paper, or people. That is the very definition of incompetent.

      Proof positive that you can work in IT and still be an idiot.

    14. Re:Last, and Dead Last by lanner · · Score: 1

      Yea, that's pretty much what he meant.

      The reality is that an entire flight scheduling system like the one that US Airways uses could probably be replaced by $50K worth of junk off of dell.com. The software has to be written custom, but this isn't computational proteomics here. A couple of SF bay goons could do this in six months.

      For this kind of a small-scale implementation, you should have at least three separate data centers across the world/continent, which duplicate the information with automatic terminal failover to the nearest operating master. You would not just want the array to be RAID, but duplicate arrays, duplicate servers, duplicate network infrastructure, duplicate entire systems three times over... and it would still cost 1/3rd or less of whatever money they are currently pouring into the black hole of incompetency that they have right now.

    15. Re:Last, and Dead Last by adri · · Score: 1

      Wow, and that likely completely underestimates the scale of this kind of project.

      Chances are there's lots of systems all tied together, feeding data in and out. There may not be a "small scale implementation". This may be a pain in the ass to integrate into lots of these inter-operating system.

      So maybe the system-as-a-whole is flawed, sure, but redesigning that to be less of a clusterfuck is likely not within the scope of your "$50k and 6 months."

      Then there's keeping everything in sync. You end up having lots of changes coming through at lots of times from lots of places. If you have backup datacentres, they also have to have copies of this data, kept in real time. If you flip to the backup, the master can't retake being master until it's brought into sync.

      If the software versions are out of date and they behave slightly differently, then you end up with fantastic and hilarious subtle issues. For example, if the software makes slightly different decisions about "stuff", then when you flip to the backup system, a whole lot of small, subtle changes in selection could snowball into much larger-scale problems down the track. Since a lot of "engineered" solutions go through periods of "design", "implementation", 10: "shit it didn't work as intended - take real life case A, B, C and implement workarounds", "workaround", "GOTO 10", you may find that these kinds of subtle issues are never really totally understood. They're worked around. Who knows what the problems are. So you flip to a backup system that has an impact on real, tangible things (such as say, luggage routing), a small change in decision making can go a freaking long way in affecting the physical world around you - and once you've started down the path of a physical-world clusterfuck, you don't always get to simply fix it with a software patch.

      Hm, what else can I think of off the top of my head. Oh right - what if the crash is because of the software + database contents? What if the crash is because the system overloaded because of some periodic job say, sweeping the database clean, or recalculating better routing rules. Chances are your backup sites will also fail - they're running the same hardware and software, with the same data? So you then say "run the backup software with an X minute delay, so if issues creep up in the primary we can stop the backups from failing the same way." The larger X is, the greater your chance of identifying problems - but the more useless it becomes when flipping over to become master. Stuff that has already been calculated by the (now failed) master hasn't been processed by the slave. So you have to integrate that data in, or design the system to cancel out the already-calculated stuff and re-issue things. Not so good if you have to cancel some real world event (eg fueling a plane) if the calculated values differ. So you choose to run primary and backups on different software versions, or revisions, or heck even independently designed systems. See paragraph 2 for the potential hilarity.

      This is why it's more difficult than $50k of junk and 6 months of goon time.

  22. Suprised? by Anonymous Coward · · Score: 1

    Given a choice between two competing airlines flying to the same place, the vast majority of passengers will book based solely on published cost.

    Leaving aside for the moment the question of hidden fees, that means that the airlines have no choice but to trim every possible cost to be competitive. I'm not in the airline industry, so I'm guessing here. I suspect that means no extra flight crews on standby for unexpected events, no extra gate crew coverage, the absolute minimum of phone lines to handle problems, etc.

    I recall flying 30 years ago, and there were airport staff sometimes standing around with nothing to do. But when things got weird, the resolution was much easier because of the flex built into the system. That was when consumers had brand loyalty (perhaps because they had no tool other than the phone to compare prices).

    On a recent trip there was a major weather event, and our plane ended up in a comedy of errors which would have been funny except for the fact that I was stuck in my seat for 12 hours. In the end, we were at an airport with no company terminals to dock at. The city run airport transportation staff had left for the night, so we couldn't be driven from the plane. According to the airport rules there, we couldn't use another company terminal unless there was an emergency. With the prospect of his passengers spending another 8 hours on the plane, the captain declared a "medical emergency" so we were allowed to deplane at the nearest empty gate. I will forever be grateful for that pilot, and the balls he had to do that, knowing that it might impact his career.

    The next morning when I tried to resolve my issue with the airline, every call to the customer service line was not met with "we regret that you will be on hold for 2 hours", but instead "we regret that we are not able to answer your call at this time", and then a dial tone. The web site offered no help either, claiming that due to system issues, they were unable to handle the volume of information requests. I ended up booking another flight on my own dime (well... the company's dime anyway) with another airline.

    Would I be willing to pay twice the cost for an airline flight to make this kind of crap go away? Sure. But I suspect I'm one of a few. I bet the vast majority of air travelers only go once in a while, and tend to forget what carrier they used last time, no matter how bad the service was.

    1. Re:Suprised? by Anonymous Coward · · Score: 0

      Given a choice between two competing airlines flying to the same place, the vast majority of passengers will book based solely on published cost.

      Leaving aside for the moment the question of hidden fees, that means that the airlines have no choice but to trim every possible cost to be competitive. I'm not in the airline industry, so I'm guessing here. I suspect that means no extra flight crews on standby for unexpected events, no extra gate crew coverage, the absolute minimum of phone lines to handle problems, etc.

      I recall flying 30 years ago, and there were airport staff sometimes standing around with nothing to do. But when things got weird, the resolution was much easier because of the flex built into the system. That was when consumers had brand loyalty (perhaps because they had no tool other than the phone to compare prices).

      On a recent trip there was a major weather event, and our plane ended up in a comedy of errors which would have been funny except for the fact that I was stuck in my seat for 12 hours. In the end, we were at an airport with no company terminals to dock at. The city run airport transportation staff had left for the night, so we couldn't be driven from the plane. According to the airport rules there, we couldn't use another company terminal unless there was an emergency. With the prospect of his passengers spending another 8 hours on the plane, the captain declared a "medical emergency" so we were allowed to deplane at the nearest empty gate. I will forever be grateful for that pilot, and the balls he had to do that, knowing that it might impact his career.

      The next morning when I tried to resolve my issue with the airline, every call to the customer service line was not met with "we regret that you will be on hold for 2 hours", but instead "we regret that we are not able to answer your call at this time", and then a dial tone. The web site offered no help either, claiming that due to system issues, they were unable to handle the volume of information requests. I ended up booking another flight on my own dime (well... the company's dime anyway) with another airline.

      Would I be willing to pay twice the cost for an airline flight to make this kind of crap go away? Sure. But I suspect I'm one of a few. I bet the vast majority of air travelers only go once in a while, and tend to forget what carrier they used last time, no matter how bad the service was.

      Shocking that it had to come to that. Did the pilot just choose a passenger or were the airport well aware and indifferent. Shocking how bad things are in the US nowadays that wouldnt even happen in Etophia

  23. CAos by Etraud · · Score: 1

    Closer to Caos... lol

  24. Re:Abhorrent Mess by Anonymous Coward · · Score: 0

    You may be right about the three-hour flight planning. I wouldn't be surprised if this were true.

  25. Give me my free stuff! by Anonymous Coward · · Score: 0

    Cheap Fares = Cheap service! What do you want for $300 (cheaper fares than 30 years ago) and if there is a problem you want REIMBURSEMENT of $500 what BS! we need to go back to real airfares and get rid of the cheapo bunch and only let the ones who can afford it fly again, Of course all the deregulation and any industry has the same results...It's Cheaper, but look where we are.
    Back to the old days and let the cheaper crowd take the bus...oh wait Greyhound cost more than airline tickets...

  26. Weather by T-Bone-T · · Score: 1

    I love the weather comments. They show that people just don't think on a large scale. I work at Dallas Love Field and I went home an hour late last night because of bad weather in the morning on the other side of the country. Most people would not realize that.

  27. No, they couldn't. by raehl · · Score: 1

    Couldn't they just look out the window? Maybe that big thing outside the window that a whole bunch of people recently walked off of just happens to be the plane everyone at the gate is waiting to board?

    Maybe, but probably not.

    If you land a plane at an airport, and go and park at a gate, what are the odds that the people waiting for a flight at that gate are supposed to be on the same plane that you just parked there?

    Even if you get lucky, how is anyone supposed to even know what gate they are supposed to be at?

    1. Re:No, they couldn't. by egburr · · Score: 1

      "If you land a plane at an airport, and go and park at a gate, what are the odds that the people waiting for a flight at that gate are supposed to be on the same plane that you just parked there?"

      Pretty good odds, actually. Every time I've been waiting at the gate when a plane arrived and disgorged passengers, that was always the same plane that we eventually got on. Except for one time they had to replace it due to mechanical problems.

      It doesn't make any sense to pull up to one gate, empty the plane, and then move it to another gate for a new load.

      --

      Edward Burr
      Having a smoking section in a restaurant is like having a peeing section in a swimming pool.
    2. Re:No, they couldn't. by egburr · · Score: 1

      "Even if you get lucky, how is anyone supposed to even know what gate they are supposed to be at?"

      It's printed on my ticket. Except for the rare last minute gate change, it's been pretty accurate.

      With the computers down, it can't be perfect, but for *most* of the flights, the available data is still good.

      --

      Edward Burr
      Having a smoking section in a restaurant is like having a peeing section in a swimming pool.
  28. and..? by CTU · · Score: 0

    The article did not really say what happened to the stranded passengers...I wish they covered that a little more.

  29. "There is another system" by WaffleMonster · · Score: 1

    The word "backup" is often confused with "practice" ... backups who needs backups?

  30. IRQ conflict maybe? by Anonymous Coward · · Score: 0

    I was recently in O'Hare passing through the terminal that houses US Airways' gates, and instead of the nice big screen LCD displays showing flight information everywhere, they had these tiny CRTs that looked like they were being run by an old PC/Jr or Amiga.

    Maybe their flight system had an IRQ conflict or ran out of extended RAM. They should really avoid loading those device drivers in the 640k base memory space. Check config.sys, perhaps?

  31. United out of Chicago, eh? by KingAlanI · · Score: 1

    I fly very rarely; last time I did was August 2010 out of Chicago (O'Hare) on United (actually a United-branded regional carrier, but it went directly back to my hometown.) Get up early to make sure I'm to the airport on time, and the flight ends up delayed several hours.

    Insufficient sample size, I know...

    * my schedule had allowed me to cheap out and take Amtrak _into_ Chicago, which went by with only relatively minor delats.

    --
    I listen to both RIAA and non-RIAA stuff if I like the music, tangential business/politics nonwithstanding.
  32. Ah, like in the dutch rail network by SmallFurryCreature · · Score: 1

    Utrecht is a central city in Holland where pretty much all the railway lines intersect for no smart reason.

    So, if there is an issue at Utrecht like a fire alarm at the control center, there better be a backup or all train travel in Holland is seriously affected.

    Luckily the backup control center is there.... right there... in the same building... small building... affected by the same fire alarm...

    But hey, lets not immidiately order a load of busses to deal with stranded passengers, people have become so used to the troubles that they will no doubt fix it themselves, yet again.

    You can't run complex operations on a shoe-string budget and expect to continue to work without a hitch. Yet we cut back on them or use budget operators because... well... we take the risk and then bitch about but will still refuse to fund public transport or book the cheapest flight possible.

    That is why there isn't a good backup solution, you are not willing to pay for it.

    --

    MMO Quests are like orgasms:

    You may solo them, I prefer them in a group.

  33. you said it: scale by fantomas · · Score: 1

    You said it: the reason people flew ok in the 40s and 50s but can't now without computers is scale. There's just so much higher volume of air traffic, critical systems depend on computers to track too many things. Random 737s flying around a crowded air space unannounced could be a little dangerous, to say the least.

    If you reduced the number of flights to 40s levels probably you could do without computers. I guess this means your regional hub reducing to flying maybe one plane an hour with 20 passenger seats on each?

  34. TIp of the iceberg... by Anonymous Coward · · Score: 0

    If we keep relying on technology to do everything for us that SHOULD have better manual / redundant systems in place, we will see more of this in every phase of our lives.

    Starliner

  35. Decentralize this. by drolli · · Score: 1

    Please, how about printing all the important information ad a 2D barcode on the tickets (cryptographically signed. of course), which can be read without a connection to the central system and having for the really important stuff = scheduling of planes a second independent system. I always wonder that the cost of these sw bugs could easily go into the tens of millions of $ , so it should be possible to take measures not to be completely dependent on a single point of failure.