Slashdot Mirror


American Airlines Grounds Flights

Sez Zero writes "The Federal Aviation Administration said American Airlines requested a halt to hundreds of its U.S. flights on Tuesday as it works to resolve a reservation system problem. American Airlines explained on their Twitter feed they had a problem accessing their reservation system. Bad day to be on the AA ops team."

91 comments

  1. Redundancy? by fustakrakich · · Score: 0

    Only in this comment...

    --
    “He’s not deformed, he’s just drunk!”
    1. Re:Redundancy? by Megane · · Score: 5, Funny

      Don't you mean "RedundAAncy?"

      It's a sAAd day for AA.

      --
      #naabhaprzrag, #sverubfr-000, #agi-fcbafberq, negvpyr[pynff*=' negvpyr-ary-'] { qvfcynl: abar !vzcbegnag; }
    2. Re:Redundancy? by DarkOx · · Score: 1

      Only in this comment...

      See now its that kinda thinking that leads to AA's entire fleet being grounded. I mean seriously you comment is a single point of failure. What if I'd scrolled past it? What is slashdot's web server failed to post your form post?

      You single comment is just the sorta cowboy IT work that causes these disasters.

      --
      Repeal the 17th Amendment TODAY! Also Please Read http://www.gnu.org/philosophy/right-to-read.html
    3. Re:Redundancy? by rwise2112 · · Score: 1

      Don't you mean "RedundAAncy?"

      It's a sAAd day for AA.

      It's AA sAAd dAAy for AA

      I hope you get modded 'redundant' to complete the joke.

      --

      "For every expert, there is an equal and opposite expert"
    4. Re:Redundancy? by isorox · · Score: 1

      Don't you mean "RedundAAncy?"

      It's a sAAd day for AA.

      It's AA sAAd dAAy for AA

      I hope you get modded 'redundant' to complete the joke.

      That was the idea, but you can never rely on mods

  2. seems to happen now and then by Trepidity · · Score: 3, Informative

    From various airlines: 2004 #1, 2004 #2, 2011 #1, 2011 #2, and probably others I missed.

    1. Re:seems to happen now and then by Anonymous Coward · · Score: 1

      And this Alaska Airlines outage from just six months ago.

      I was actually in the line pictured there at PDX. But, I left before the photographers arrived to re-book on another airline. It's really unfortunate I was able to re-book. That outage was an answer to my prayers on the way to PDX for something to happen to get me out of going on that business trip.

    2. Re:seems to happen now and then by mjwx · · Score: 1

      From various airlines: 2004 #1, 2004 #2, 2011 #1, 2011 #2, and probably others I missed.

      OK, I'm travelling to the US later this year, I'm going to have to take some internal flights... Is there any airline there that I can reasonably count on not to screw up?

      --
      Calling someone a "hater" only means you can not rationally rebut their argument.
    3. Re:seems to happen now and then by Anonymous Coward · · Score: 1

      No. You're fucked mate.

  3. is it because of this? by Anonymous Coward · · Score: 0, Interesting

    http://www.tnooz.com/2012/06/12/news/all-bets-are-off-american-airlines-to-abandon-hp-reservations-project-for-new-partner/

  4. Are they on some older software that can't hand mo by Anonymous Coward · · Score: 0

    Are they on some older software that can't hand more then X changes in X time?

  5. Move the planes from the gates by magarity · · Score: 4, Informative

    "At American's hub in Miami, The Miami Herald reports that landing AA flights have run out of available gates since none of the airline's departures are taking off. A passenger on one of those flights -- 66-year-old Richard Bell -- tells the Herald he had been stuck on an AA flight arriving from Baltimore. He told the newspaper that the aircraft's engines were running and that the air conditioning was working. But he also said the flight's pilots come over the public address system to warn fliers that some other systems were not functioning. "He mentioned the toilet specifically as a problem,'' Bell tells the Herald."

    This is total lack of human compassion that someone can't get in one of those tractors, push the plane at the gate out of the way to a spot off to the side and let the plane with the people unload. What kind of heartless ass is running American's operations at that airport? Oh, gee, that might inconvenience the airline personel because the first plane would then have to be trundled back over since it needs to leave first when things resume.

    1. Re:Move the planes from the gates by Farmer+Pete · · Score: 2

      I agree, but I think the solution could be a lot easier. Most airports have stairs they can drive to the airplane door. Sure, that wouldn't help a disabled person, but there is no reason 200 people have to sit in a plane for hours when only 1 or 2 people can't walk down a flight of stairs.

    2. Re:Move the planes from the gates by nblender · · Score: 1

      I thought there was some sort of protocol that says "when a passenger gets off the plain, their luggage must also get off the plane"? If so, that would mean they'd have to completely undo the flight and then redo it when it was ready to go...

      Not sure about this.

    3. Re:Move the planes from the gates by mrbester · · Score: 1

      If the passenger is still airside there is no reason to remove baggage, especially if it is an organised deplane to a holding area (because you aren't getting landside again until cleared) with an expectation of the flight taking place. Should it then be cancelled the usual routine of removing baggage would take place, just like with any arrival.

      --
      "Wait. Something's happening. It's opening up! My God, it's full of apricots!"
    4. Re:Move the planes from the gates by magarity · · Score: 2

      The stoppage in question was a "ground halt" meaning that once the planes get where they're normally going, they don't leave again. So the passengers are where they're supposed to be; OK, why not take their luggage off? Then shove the plane out of the way and get the next one into place that's otherwise idling full of people wasting their lives waiting. Maybe a handful of people are continuing on to that same plane's next destination but that's a really low percentage of the passengers.

    5. Re:Move the planes from the gates by couchslug · · Score: 3, Informative

      It's fucking laziness.

      I've towed MANY aircraft and it isn't at all difficult to do.

      Basically, back towbar up to nose gear, unpin steering links (your towbar does the "steering" when towing), connect towbar (aircrew remain in cockpit to apply aircraft brakes if towbar accidentally disconnects under tow), remove wheel chocks, tow aircraft to new spot, chock wheels, disconnect towbar, reconnect steering links, drive tug to next job.

      It "ain't shit" be the aircraft large or small.

      --
      "This post is an artistic work of fiction and falsehood. Only a fool would take anything posted here as fact."
    6. Re:Move the planes from the gates by mjwx · · Score: 1

      This is total lack of human compassion that someone can't get in one of those tractors, push the plane at the gate out of the way to a spot off to the side and let the plane with the people unload. What kind of heartless ass is running American's operations at that airport? Oh, gee, that might inconvenience the airline personel because the first plane would then have to be trundled back over since it needs to leave first when things resume.

      Not quite that simple as you'll need an excavator to move all the support infrastructure (fuel, sewerage inlets and so forth).

      What the airport should have is an over-flow area on the tarmac where passengers can be unloaded via those portable stairs you may have seen about and bused to the terminal. I've seen this setup in almost all SE Asian airports except for NAIA in Manila (and I'm not sure if that is because NAIA was designed by Americans or run by Filipinos).

      This of course wont help with flights departing late.

      --
      Calling someone a "hater" only means you can not rationally rebut their argument.
    7. Re:Move the planes from the gates by Anonymous Coward · · Score: 0

      That was after 9/11 and known as positive bag match. Shortly after TSA took over baggage screening, airlines have no longer been required to remove baggage when that passenger doesn't fly.

    8. Re:Move the planes from the gates by mjwx · · Score: 1

      I agree, but I think the solution could be a lot easier. Most airports have stairs they can drive to the airplane door. Sure, that wouldn't help a disabled person, but there is no reason 200 people have to sit in a plane for hours when only 1 or 2 people can't walk down a flight of stairs.

      In addition to the stairs, they also have elevators on trucks (this is how they get large items in and out of the cabin).

      --
      Calling someone a "hater" only means you can not rationally rebut their argument.
  6. Re:Are they on some older software that can't hand by Trepidity · · Score: 3, Insightful

    The software they run is Sabre, which was co-founded by American Airlines some decades ago. I have no particular knowledge of which software has undergone rewrites and which hasn't, but if you scan their own timeline, it's not hard to suspect that there are huge piles of ancient code still in there.

  7. Um...any TECHNICAL explanation? by xxxJonBoyxxx · · Score: 4, Insightful

    Kudos, SlashDot, for getting the story here on the same day as the rest of the media. Now how about some links that AREN'T ConsumerNews or USAToday or other crap. Does anyone know what the TECHNICAL reason for the failure is?

    1. Re:Um...any TECHNICAL explanation? by Anonymous Coward · · Score: 0

      Does anyone know what the TECHNICAL reason for the failure is?

      Nope so I'll make something up... It was a track "hoe" or one of many copper thieves who lack requisite number of brain cells to know the difference between fibre and copper.

    2. Re:Um...any TECHNICAL explanation? by JustOK · · Score: 2

      Technically, there was something that went wrong, probably something to do with a technical problem with some of the technology. Technically, that's just a theory, 'tho.

      --
      rewriting history since 2109
    3. Re:Um...any TECHNICAL explanation? by zachary.grafton · · Score: 2

      I'd imagine it's because no one wrote any documentation when the software was created, and none of the airlines have anyone dedicated to tracking what documentation was written. So, everyone is standing around looking at a terminal trying to figure out what's going on with no idea where to even begin.

    4. Re:Um...any TECHNICAL explanation? by Anonymous Coward · · Score: 0

      I can be a media outlet for you with mainstream caliber reporting:

      An official that wishes to remain anonymous stated that the likely cause for the failure is presumably a paperclip that has made a key stuck on a keyboard. Some industry experts, however, considered such a cause impossible although remained open to the possibility that there was coffee on a keyboard involved.

      Good enough for you?

    5. Re:Um...any TECHNICAL explanation? by Anonymous Coward · · Score: 0

      Kudos, SlashDot, for getting the story here on the same day as the rest of the media. Now how about some links that AREN'T ConsumerNews or USAToday or other crap. Does anyone know what the TECHNICAL reason for the failure is?

      Your mom sat on the server. It quickly gave out and was rendered flat as a pancake.

      Seriously, how would anyone here know something the media doesn't? /. has an inside track on the inner workings of AA's IT systems? stfu and get over yourself.

    6. Re:Um...any TECHNICAL explanation? by EETech1 · · Score: 1

      +2
      That's so true it's funny.

    7. Re:Um...any TECHNICAL explanation? by Anonymous Coward · · Score: 0

      My guess is some sort of communications cable failure. I forget which airline, but one of them had both redundant fiber optic cables cut in the same day.

    8. Re: Um...any TECHNICAL explanation? by hemp · · Score: 1

      Actually, it is a very well documented system with strict change control and change management.

      --
      Skip ------ See the latest from http://www.anArchyFortWorth.com
    9. Re:Um...any TECHNICAL explanation? by Anonymous Coward · · Score: 0

      Second hand information from someone in the office trying to book a flight today: Apparently if you tried to select your seat prior to buying your ticket, the system would delete your reservation after your paid for your ticket. He thought he had three or four duplicate charges he was going to have to get reversed. Dunno if that is really what the failure scenario was or if that was just a side effect of the disconnect between the web front end and the very old and decrepit back end.

  8. Re:CmdrTaco's Fault by turkeyfeathers · · Score: 0

    Doesn't CmdrTaco still get royalties everytime a Packt book review is published here?

  9. Disingenuous headline by Anonymous Coward · · Score: 1

    Please don't use headlines to get ad impressions.

    You know very well what people's first reaction will be to that headline, coming a day after the marathon attacks.

    How about this: "American Airlines IT Problem Grounds Flights". Still shorter than the average Slashdot headline.

  10. Re:Are they on some older software that can't hand by xxxJonBoyxxx · · Score: 3, Funny

    >> piles of ancient code still in there

    ancient code ain't always slow code - remember what we had to write it for, you whippersnapper

  11. Re:Are they on some older software that can't hand by zachary.grafton · · Score: 5, Informative

    I currently work for Silver Airways, but as I've experienced first hand, almost all of the airlines software system's suck some major ballsack. None of the airlines have undergone the massive rewrite that their reservations system need and they just keep bolting on more pieces of crap code to these ancient systems. United uses SHARES, and so does Silver Airways, but 2 different versions of the same software, and to make things worse, ours is rented off of Island Air and not a single human being can fix any of the stuff they need to do. The Silver Airways version doesn't even handle baggage charges in a decent fashion, forcing their employees to build a reservation just for a bag fee. It's always great when you have a functional TELEX printer that you can't use because not even the IT department can figure out how the hell SHARES is trying to route output. Personally, all the airlines are damned for not spending any money in updating reservations systems from the late 70s and early 80s. If you anyone is interested, you can basically trace all of the airlines reservations systems back to a company called EDS, which was at one time owned by Ross Perot.

  12. I've seen worse excuses for flight problems... by FooAtWFU · · Score: 1
    --
    The World Wide Web is dying. Soon, we shall have only the Internet.
  13. Cover story? by Frank+T.+Lofaro+Jr. · · Score: 0

    Perhaps they had intelligence on a threat to an American Airlines flight and didn't want to alarm people...

    al Qaeda usually does more than one attack at a time we've seen.

    --
    Just because it CAN be done, doesn't mean it should!
    1. Re:Cover story? by MouseTheLuckyDog · · Score: 1

      OR maybe itw as AQ hackers bringing SABER down.

    2. Re:Cover story? by matrim99 · · Score: 4, Funny

      I think that a passenger saw a monster on the wing, like what happened in "Nightmare at 20,000 Feet".

      Hey, if we're going to be making up random shit, it might as well be something cool.

      --
      Right. No, your other right. No, the other other right.
    3. Re:Cover story? by Anonymous Coward · · Score: 0

      There's some men in white coats here to see you...they came in their own vehicle...

    4. Re:Cover story? by LifesABeach · · Score: 1

      I thought AA was "to big to go down," on its SABER.

    5. Re:Cover story? by LifesABeach · · Score: 1

      That passenger would go on to command a Star Ship...

  14. Re:Are they on some older software that can't hand by Trepidity · · Score: 1

    True. A bigger problem is probably that it's 50 years of accumulated cruft that may or may not work together in any kind of maintainable way (probably "may not"). Stable legacy systems that are just maintained with minor bugfixes now and then can be perfectly reasonable, but systems that accrete code over decades tend not to be.

  15. Re:Are they on some older software that can't hand by TsuruchiBrian · · Score: 2

    old code is plenty fast. It's just full of bugs, non-scalable, unmaintainable, and platform dependent. When we had slow computers speed was the only thing that mattered. Now we have fast computers and maintainability, scalability, and platform independence is worth a few lost clock cycles.

    Old code is fast, but not if you count the time it takes to write, maintain, rewrite, and all the time spend debugging and rebooting computers and restarting services when things don't work.

    Brian Kernighan put it eloquently: "Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?"

  16. More Important by Anonymous Coward · · Score: 0

    Bad day to be an AA customer.

  17. Re:Are they on some older software that can't hand by RabidReindeer · · Score: 2

    Sabre was one of the original mainframe transaction processing systems. I can run OS/MVS under the Hercules System/370 emulator on one of the old junk Pentium 166 boxes sitting over in the corner and it would still out-perform the original Sabre computers by a considerable margin.

    On the plus side, programs back then didn't have to deal with 16 different UI controls (menu, popup menu, toolbar, command keys, etc) so the source code base was much smaller. And you never rebooted a mainframe unless the world was coming to an end, almost literally. Certainly not for application bugs.

    On the minus side, that stuff was never intended to interface in all the myriad ways that systems routinely do now. Which means that support for Internet users and modern GUIs was all bolted on as after-the-fact stuff. Probably by cheap outsourced labor.

    This isn't the first recent Sabre problem, just one of the more severe ones.

  18. Re:Are they on some older software that can't hand by TsuruchiBrian · · Score: 1

    The real issue is that these airlines are usually publicly owned. They care about profits and rewriting all the infrastructure code is expensive and might even stop normal operation (resulting in higher costs). When shareholders do the math for what makes most financial sense, it is usually to keep sticking on more duct tape until duct tape no longer works. At some point a full rewrite makes the most financial sense, and that's when it will happen.

    Even when an airline is just stupid and only wants duct tape forever. All it takes is for one airline to do the rewrite and reap the benefits of better efficiency. That airline should in theory be able to have much more efficient scheduling of resources and less downtime. This allows them to make higher profits and/or undercut their competitors. Other airlines will see the benefits and follow suit. The ones that can't update their software will be consumed by the ones that can.

    On the other side of the spectrum, constantly rolling out infrastructure rewrites is pretty inefficient as well. There is some optimal lifecycle for these sorts of things (e.g. 20 years ?). Too far over or under, and you are not making as much profit as you could have been.

  19. Technically it's a theory. by Anonymous Coward · · Score: 0

    But technically correct.

    1. Re:Technically it's a theory. by sconeu · · Score: 1

      But technically correct.

      The BEST KIND of Correct!!!!

      --
      General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.
    2. Re:Technically it's a theory. by JustOK · · Score: 1

      I was literally trying not to speak figuratively

      --
      rewriting history since 2109
  20. Patched without testing by Anonymous Coward · · Score: 0

    How much you want to bet?

  21. Re:Are they on some older software that can't hand by zachary.grafton · · Score: 2

    This is pretty much what is killing it. I've made a post about SHARES already, which is a United system, but it seems to be pretty popular to use old systems in the airlines and I imagine SABRE is in a similar state. SHARES was originally just a reservations system, but they've tacked all sorts of modules on to it. For instance, SHARES has an extension for WorldTracer(for tracking bags,) a cargo tracking module, a module called FOMS for flight tracking, and all sorts of other cruft. They've built some GUI wrappers for some bits and pieces, but none of them are worth mentioning. So, most ticket agents are still doing everything through an archaic command line interface to a mainframe. The software also does more things in the core module above and beyond just reservations too, like flight information so gate agents can track flight statuses. Personally, I think they are going to run into a wall with the extensibility of the system in the next 5 years and a lot of airlines are going to be playing catch up with their IT systems. Most of them just need to be redesigned from scratch. It would probably make for a better customer experience and make it a better workplace for their employees.

  22. Re:Are they on some older software that can't hand by kwbauer · · Score: 1

    Since when does any shareholder ever decide what internal projects get worked on?

    Since "the real issue is that these airlines are usually publicly owned" then the solution is to have government announce regulations that all airlines must be owned by a single person? Or maybe family owned or even some partnerships? That will solve the problem because you won't have a bunch of nameless, faceless people begging for a larger quarterly payout (dividend) but will instead have one person (or a handful of people) demanding a larger quarterly payout (profit)..

  23. Security Theater by Anonymous Coward · · Score: 0

    AA sells front row seat today and tomorrow and the next.

  24. Re:Are they on some older software that can't hand by Sponge+Bath · · Score: 2

    The may have tried it on Windows 8 for the first time and could not find the Start button.

  25. Re:Are they on some older software that can't hand by mcmonkey · · Score: 1

    Brian Kernighan put it eloquently:
    "Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?"

    I'm going to guess that was said tongue-in-cheek. First, writing and debugging are two different, although related, skills. While writing (of code or anything else) does benefit from independent review, I doubt Mr. Kernighan really thinks there is nothing to be gained when someone reviews their own work.

    Second, I can debug my own code because I'm smarter when debugging than I was when writing. I have the benefit of the experience of writing the code and observing the results of running in a test or sandbox environment.

    Third, I hope Mr. Kernighan isn't seriously suggest folks try to be "as clever as you can be" when writing code. Clever is nice for showing off, such as for obfuscated code contests. For the real world, where code has to be tested, documented, and maintained, clever should be avoided. Give me clean, correct code over clever.

  26. Re:Are they on some older software that can't hand by Forever+Wondering · · Score: 1

    [According to the article] there was nothing wrong with the software. It was that the network couldn't be accessed [was down]. This sounds more like a core router failure or a router that polluted the routing tables.

    Airline reservations systems fall into the "mission critical" software development category. They tend to have fewer bugs because they get so much testing before deployment.

    And one of the reasons AA spun off its [in-house] Sabre system was so that it could be kept modern. Another reason, IIRC, was that when it was in-house, they were being accused of antitrust, because other systems couldn't access it. At the time, the only serious competitor was United's(?) Apollo system.

    In my experience [of 40 years as computer engineer], fast code actually has fewer bugs, because, it order to make it fast you find the simplest way to state the problem. This simplicity [or elegance] tends to keep the code clean, maintainable, fast--all at the same time. When you don't keep an eye on performance, your code can become convoluted (i.e. slow) since you don't care about performance, you can get sloppy. That's when bugs creep in.

    --
    Like a good neighbor, fsck is there ...
  27. Re: Are they on some older software that can't han by Anonymous Coward · · Score: 0

    Hey Sabre employee here. Yes that is what happened. We put a ticket on stackoverflow but no working solution yet.

  28. Re:Are they on some older software that can't hand by Anonymous Coward · · Score: 0

    @RabidReindeer : right on !! ...being on the main frame side here for a canadian airline , this type of crash is not workstations/OS related ( despite being true that raw data must now feed a bunch of newer needs/software ) , and yes, even if planes have stairs to get in , its not enough to handle all duties an airline must face

  29. Re:Are they on some older software that can't hand by Anonymous Coward · · Score: 2, Informative

    I'm a Sabre employee. The issue today was not related to any Sabre systems.

    “American Airlines mistakenly reported they were having an issue with the Sabre reservations system, which they subsequently corrected. To clarify, all Sabre systems are operating as normal and have been all day. We see American Airlines is now up and running. We stand ready to help if needed.”

    Dan

  30. Re:Are they on some older software that can't hand by TsuruchiBrian · · Score: 1

    The shareholders are the collective owners of the company. They are the boss. They can have as much influence on what internal projects get worked on as they want. By deciding not to be involved, they are making an implicit decision to allow this to be handled by people lower in the totem pole. The owners typically decide on budgets which either allow for total rewrites or not. The engineers/engineering department can ask for more money to do a good job, and these requests are either approved or denied at a higher level.

    Sometimes the shareholders don't care about anything until it becomes a problem that affects profits. Sometimes they get involved sooner if they have a larger interest in the company succeeding.

    Small shareholders typically do not participate in votes or they can grant their voting power to other shareholders.

    No not every single shareholder can have a say in the day to day operation. But in aggregate they are the highest entity on the totem pole. They have the power to decide to do or not to do a software rewrite if they choose not to abdicate this power. The managers, even the CEO, is just executing the will of the shareholders. They don't have the power to authorize a software rewrite without the approval of the shareholders, if it is going to be a large expense (at least not without negative consequences).

    If doing a software rewrite is somehow a small expense (i.e. fits in the allowable budget), then doing it is no problem for anyone, so it will probably get done.

  31. Re:Are they on some older software that can't hand by TsuruchiBrian · · Score: 1

    The point of his statement was that software should be written in a way where it is easy to understand and maintain. It is much easier to make "hard to understand code" than it is to understand "hard to understand code".

    If you are smarter when debugging, then you are already following his advice, because you are not "clever as can be" when writing it, because you are less clever writing it than when you are debugging it.

    He is suggesting you *shouldn't* be "as clever as can be" when writing code, *because* it will be much harder to debug.

  32. Re:Are they on some older software that can't hand by TsuruchiBrian · · Score: 1

    You can write super fast code that uses goto, global variables, etc. Yes you can write very slow code that's full of bugs but that's not the point. My point is that the fastest code is going to be more prone to having bugs. The easiest to maintain code is going to be slightly suboptimal (slower than the fastest code).

  33. Re:Are they on some older software that can't hand by BitZtream · · Score: 1

    The shareholders are the collective owners of the company.

    You, with your 4 shares of stock don't get to tell anyone anything.

    However, when you own enough shares that your vote matters to a large enough extent, you don't get to tell them exactly what to do ... but you sure as hell can make someones employment at the company difficult to impossible.

    The reality of it is however, 99.99999% of the time, a total re-write is a retarded idea. You just get a whole new set of bugs and you've wasted all the time you put into the original system. There is nothing that actually prevents old code from being modernized and cleaned up, it just takes actual effort and attention to detail ... and that is why so many people think re-writing it is the solution, they are lazy and/or are not qualified to do the job with the required attention to detail. Instead they rewrite it, and IF it ever gets back to doing all the stuff the original did, its generally ten times more bloated and slower still as now you've just had commitees telling you how to use a new design to build something that was already optimized and well understood by its users.

    Anyone suggesting re-writing large systems in their entirity is almost certainly unqualified to be making any such sort of recommendation.

    --
    Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
  34. Re:Are they on some older software that can't hand by Zmobie · · Score: 2, Informative

    You see, no it isn't exactly like you say. I have written and have in production interfaces with SHARES and the problem isn't so much their infrastructure, as they are actively doing a lot more upgrades (I actually have had conference calls with their IT managers while migrating one of their systems) than anyone thinks. The problem is more operational than anything, but since I am actively under an NDA concerning things like that I can't bring up any specifics. Other than that, the other problem is too many vendors under the same roof. I have seen systems that have had probably every company in the business do an update somewhere to some module or piece of the system and THAT is what makes these things more of a nightmare than anything. The nice simple ones where it was a streamlined job, are very easy to upgrade and keep current.

    Now that isn't to say that some of the current systems are not old and outdated, but many of the carriers are in the process of performing these upgrades right now (I have insider knowledge on that because of the work my company does). The general time-frame for large scale system upgrades varies between 5 to 10 years it seems, depending on how stable the original system actually was. I can also say that American actually just rolled out some new updates to several of their systems including their check-in and processing, and they are currently doing upgrades to other existing framework elsewhere. Smaller upgrades are usually much more frequent, but these are more behind the scenes and not something a customer would ever see.

    Most of the legacy systems are surprisingly stable, and in fact there are more hiccups during the initial roll-out of new systems than anything else. This one is still up in the air it seems as to what or who was the culprit. More than likely it is a new system hiccup and things kind of fell through at a really bad time. I feel for whoever was scrambling trying to fix it, I have heard some nasty stories from our guys on emergency support calls and have had a few crappy ones myself.

    And FYI you are referring to FIMS not FOMS, and United actually uses FLIFO for their flight information (again have interfaces in production right now working with it), and SHARES is run through mostly data-center grade servers and not mainframes (though they do still use a lot of command line to interface with the system).

    P.S. I don't mean the post to come off condescending or anything, just I have much more intimate knowledge of these systems and felt I could share a little bit to allow for a better understanding of things.

  35. Re:Are they on some older software that can't hand by TsuruchiBrian · · Score: 2

    Owning 4 shares in a company gives me about as much say as a voter in an election. No I don;t get to decide who the senator of my state is, but I do have influence over it, even if that influence is small. All the voters added up completely determines the outcome. It's not important that the smallest shareholder always affects the outcome of every decision, it only matters that this is possible. Just like I don't need to cast the deciding vote in an election for my vote to count.

    It depends on your definition of a *total* rewrite. Does every single line of code need to change for it to count as a rewrite? It is very likely that the number of lines changed will be > 0 and less than 100%. If you change 95% of the code, that's pretty much a rewrite in my opinion.

    Yes changing software always introduces the potential for new bugs. Lots of software is written by and used by only one company. This software even if used for a very long time is not necessarily well tested. A rewrite of a companies database code could consist of ditching a custom database and replacing it with an enterprise database which works better and has been tested more.

    Is this a rewrite? Well it is in the sense that almost none of the old code is still there. It isn;t in the sense that most of the new code is actually existing and well tested code.

    I do lots of "rewriting" for my job. I take chunks of old code that are 15 years old and full of bugs that have never been caught, and port them to Qt usually reducing the size of the code by a factor of 10 and leveraging the fact that Qt has platform independent classes for stuff like ByteArrays, TCP sockets, Linked Lists, DataStreams, etc. This not only allows the code to run on 64 bit targets, but also fixes a bunch of problems. There really isn't any reason to keep custom written buffer or network logic that was kludgey to begin with. If I end up changing 1% or 99% of the lines of code then so be it. I usually end up in the 90% range.

    Usually you don't want the new code to do everything the old code did. You want the new code to do what the old code was supposed to do.

  36. I Think They Were Cracked... by LifesABeach · · Score: 1

    I've been getting daily confirmations of tickets to places. It's not impacting my credit card. A bank spokeswoman called and said someone had gotten our PIN number and was making the bogus reservations. Funny thing, I and my wife never use our debit card for purchases. Given AA's track record for lost luggage, broken items in luggage, with liberal DHS stickers on the breakage, the crap service at LAX. Maybe someone got feed up, and then smiled?

  37. Re: Are they on some older software that can't han by DeathElk · · Score: 1

    Perhaps you should log a query on expert sex change

  38. Re:Are they on some older software that can't hand by starblazer · · Score: 1

    I'd rather type 4 lines of code to rebook your ticket and resync rather than have to wait for AirportApps or whatever GUI to refresh and do it. Six if you include the "ER, ER" to finish the job.

    Now, would I want to do a new PNR in SHARES? absolutely not. but for day to day operations, SHARES isn't bad. If you want to see cruft, try working with Deltamatic. The same thing that takes one command in shares takes 3-4 in Deltamatic.

  39. Re:Are they on some older software that can't hand by starblazer · · Score: 1

    They are stable, I'll give you that, but I wish that I could get through a day without having to reset my terminal a few hundred times because I get stuck in a terminal with "X-Wait" and a locked term.

  40. Re:Are they on some older software that can't hand by jelizondo · · Score: 1

    What is missing from the timeline is the time they blew up a huge project. I remember reading about it in the early 90's but can't locate the source, so I give you Wikipedia: basically they blew 125 million and 3.5 years of development work and AMR (American Airlines parent company) was sued by Marriot, Hilton and Budget (partners in the system) for the failure.

    Then back in 2009, AMR hired HP to develop a new system for them, which was seen as very risky.

    Now, it seems that they have thrown in the towel, what is it with AMR that it can't get a fucking system going after 20-something years?

    --
    Be very, very careful what you put into that head, because you will never, ever get it out. - Cardinal Wolsey
  41. Re:Are they on some older software that can't hand by Forever+Wondering · · Score: 2

    I'm a [linux] kernel programmer. I write device drivers. I haven't used a goto in 30 years [not one!]. I've also written code to do realtime processing of broadcast quality HD video on a linux platform in conjunction with specialized hardware.

    You're assuming that to make code fast, it's done [has to be done] by doing insane hacks [or the bad practices that you mentioned]. This [usually] only produces modest speedups on the order of a few percent. Most really fast code is highly modular and just uses a better algorithm (e.g. instead of linear search, use binary or RB tree, etc.).

    As the simplest example I can think of:

    If you use:

        int a[100];
        int idx;
        int sum;

        sum = 0;
        for (idx = 0; idx < 100; ++idx)
            sum += a[idx];

    You're better off with:

        int a[100];
        int *ptr;
        int *ep;
        int sum;

        ptr = a;
        ep = ptr + 100;

        sum = 0;
        for (; ptr < ep; ++ptr)
            sum += *ptr;

    When such loops become slightly more complex, the latter will optimize better and run 2x faster. That's [culled] from a real world example, where I recoded along those lines. And the resulting code was simpler. I also moved out loop-invariant expressions that the optimizer didn't catch. No "tricks" [if you will] were needed.

    At the video company, we had to load firmware into an FPGA. The original loader program [written by the FPGA company] took 15 minutes. That was intolerable for our customers. But, I discovered it was using fscanf on the input file for each and every input byte (e.g. fscanf, send to device, repeat). This was grossly inefficient, but was never changed because the code was originally designed for very small FPGAs that were only sparsely filled (e.g. the slow code would take about 2-3 minutes for most of the FPGA company's customer use cases). However, we were using the largest FPGA possible and filling it to capacity.

    I recoded this by preloading the entire file into a memory array, transforming it, so that it became:
        for (all_bytes_in_array)
            sendbyte();
    This reduced the load time from the 15 minutes to 90 seconds [a 10x speedup]. Once again, the resulting code was far simpler than the original. Oh, and the original code was sparsely commented. I added comments to virtually every section [as part of my original investigation, before changing any code--so I could understand things well enough to change the parts that I did change]. In so doing, I found multiple redundancies that could be eliminated. This is more Occam's Razor than anything else.

    I've got a friend at a company that is using Scala ("Java done right" (tm) :-). Based on the horror stories he's told me, the programmers there that have the most bugs filed against them are the ones that are the strongest proponents/users of Scala's "functional programming" features. Ironic to say the least.

    On the academic front, Carnegie Mellon [arguably one of the top three schools for computer science], is changing its freshman curriculum. They will no longer introduce Java, will eschew OO and functional programming in favor of a more [traditional] imperative programming approach [with increased emphasis on performance analysis and details of algorithm implementation]. Historically, CM has usually been a trend setter, so that should give one some food for thought.

    --
    Like a good neighbor, fsck is there ...
  42. They actually corrected themselves... by Anonymous Coward · · Score: 0

    The issue was not the reservation system, it was American Airline network having issues accessing Sabre, they corrected their tweets

  43. If you don't pay . . by Anonymous Coward · · Score: 0

    This has "failure to pay your Sabre bill" all over it !!

  44. Re:Are they on some older software that can't hand by drinkypoo · · Score: 1

    On the plus side, programs back then didn't have to deal with 16 different UI controls (menu, popup menu, toolbar, command keys, etc) so the source code base was much smaller.

    But IBM terminals do support things like menus, toolbars, command keys, text input fields, and the like.

    --
    "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
  45. Re:Are they on some older software that can't hand by Chriscypher · · Score: 1

    Owning 4 shares in a company gives me about as much say as a voter in an election. No I don;t get to decide who the senator of my state is, but I do have influence over it, even if that influence is small. All the voters added up completely determines the outcome. It's not important that the smallest shareholder always affects the outcome of every decision, it only matters that this is possible. Just like I don't need to cast the deciding vote in an election for my vote to count.

    Have you ever actually voted your proxy? It does not work like a government election.

    You can vote or withhold your vote for a select list of directors. They are more often than not the current board of directors. You do not get to choose between contenders for the same board seat. Since the directors on the ballot are nominated by shareholders with far more shares than you, your vote is mostly symbolic and meaningless.

    Unless you are CALPERS, manage a large trust or investment fund, or are otherwise a .1%-er with a heavy stake, you cannot dictate board nominees, and therefore have no say in oversight.

    Bottom line: You can vote for or against, but do not get to choose your oligarchs.

    --
    "You have liberated me from thought."
  46. Re:Are they on some older software that can't hand by RabidReindeer · · Score: 1

    On the plus side, programs back then didn't have to deal with 16 different UI controls (menu, popup menu, toolbar, command keys, etc) so the source code base was much smaller.

    But IBM terminals do support things like menus, toolbars, command keys, text input fields, and the like.

    I'm not sure what IBM terminals you are talking about, but I'm referring to "green screen" terminals. Their idea of a menu/toolbar/commandkey UI was to have a row down at the bottom of the screen that said "F1 Help F3 END F7 FORWARD F8 BACK".

  47. Re:Are they on some older software that can't hand by mcmonkey · · Score: 1

    He is suggesting you *shouldn't* be "as clever as can be" when writing code, *because* it will be much harder to debug.

    Oh. That I agree with 100%. I didn't really understand his point the first time I read the quote. Thank you.

  48. Re:Are they on some older software that can't hand by zachary.grafton · · Score: 1

    I'm not suggesting a rewrite of the code based on it's maintainability or the amount of effort required to fix bugs. I suggest a rewrite because a lot of the concepts used in these systems are antiquated in and of themselves. Not to mention training people on systems like this is a huge undertaking because of the way the environment has been modeled, and a lot of times, trying to change the concepts behind a large bulk of code just doesn't work out to be as cheap as a whole rewrite. These systems aren't well understood by its users and that is a problem. You can't pick people up off the streets and pay them the typical airline pay and expect that type of person to learn and use these systems effectively. Having said that, I actually enjoy using a lot of these systems (most of my coworkers find them scary and painful to use), but I would definitely like a lot more consistency across the various programs that the airlines expect their employees to use. I don't know if a whole rewrite is required, but from an end user's perspective, some really big changes need to be made.

  49. Re:Are they on some older software that can't hand by TheLoneGundam · · Score: 1

    "X-Wait" doesn't mean it's never coming back... if you notice it, it means that response time is slower than you expect but it still might come back. Many mainframe block-mode terminals and application "servers" like CICS (CICS is most analogous to Tomcat though the comparison isn't exact) "lock" the terminal until the transaction response is produced, to keep people from entering hundreds of transactions in a row and losing track of which response goes with which transaction. So, if a transaction is delayed for some reason, your terminal is still "locked". Impatient people don't like the situation, reset the terminal emulator, and enter the transaction again - but the only effect of that is to throw away the result of the first transaction when CICS goes to send it (because the terminal session was lost) and put your transaction at the back of the queue. That's not to say that things _never_ get hung up in the locked state, of course there are failures where the response never comes back, but it's good to pick some amount of time (30 seconds, 1 minute) to wait before going through the "throw it away and start over" process.

  50. Re:Are they on some older software that can't hand by TsuruchiBrian · · Score: 1

    You are saying that often a program *can* be made faster by also making it better. I am not disputing this. The most obvious example of this is choosing an algorithm that is asymptotically efficient.

    But even once you get to the point where you wouldn't change a single character, there are usually still things to can do to increase speed but at the cost of modularity, scalability, maintainability.

  51. Re:Are they on some older software that can't hand by TsuruchiBrian · · Score: 1

    Let's say a company has 1 million shares. I have one share. I don't have much influence. If I collectively work with half a million other people with once share, then we have the power of someone with a 50% stake in the company.

    This is very similar to government elections except that some people get more than once vote. That doesn't change the fact that someone with 1 vote has only a small amount of influence.

    No I have never owned stock. I spend all my money on my mortgage. I don't doubt that most investors with small shares bother proxy voting. Maybe they feel it is not worth their time to do the research to make an informed decision that would affect them. That doesn't mean that they can't participate if they wanted to. Lots of people don't bother voting in political elections either for the same reason.

  52. Re:Are they on some older software that can't hand by drinkypoo · · Score: 1

    Their idea of a menu/toolbar/commandkey UI was to have a row down at the bottom of the screen that said "F1 Help F3 END F7 FORWARD F8 BACK".

    There's incredible variation in "green screen" terminals. An adm3a doesn't even have cursor control. An IBMwhatever has fillable fields and limited form validation. A Tek terminal draws graphics. All of these are just a terminal, though the Tek may have a mouse it's still not an X terminal.

    --
    "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
  53. Re:Are they on some older software that can't hand by RabidReindeer · · Score: 1

    Tek, LSI, ADDS, and similar terminals aren't IBM terminals, by definition.

    Back when Sabre was new, the IBM mainframe spoke strictly to IBM terminals - either the 3270 series or their predecessors (2660). I spent quite a few years working in that world. As befitted a mainframe, they transmitted data in blocks, not per-character like their ASCII cousins. They didn't have validation as such, just the ability to lock out non-numeric input to numeric fields and stuff like that. Definitely no GUI widgets except what you could draw with "ASCII Art" (technically, it was EBCDIC Art).

    Attaching a non-327x terminal to an IBM mainframe back then was not trivial. The 3277 terminals were practically hollow shells, with what little actual intelligence they had emanating from the 327x control unit, which is what actually talked to the mainframe itself.

    Sabre is so old that for all I can recall it originally debuted on 2660 terminals. But I don't think it supported anything more powerful than a 3279 up until PCs started taking over as terminals.